The Case of the Picky Org Pick Lists
Summary
A case study on a development process using OSIDs. A service architect interferes with an application project and an OSID implementor.
The Provider To All People
A Resource OSID Provider for Organizations
An OSID implementation programmer is tasked with creating a simple Resource OSID Provider to reference the organizations at some hypothetical university. He is told that it will be a valuable service that could be used in a variety of situations. He pulls the data from the Org table in the data warehouse.
The output looks like:
ResourceLookupSession.getResources() ==> Parking Food Technology Office of Sponsored Research Chemistry Electrical Engineering & Computer Science Management Physics Architecture Skydiving Club Civil & Environmental Engineering Humanities Textile Technology Urban Studies & Planning Facilities Cancer Research Center Earth, Atmospheric & Planetary Science Railroad Engineering Linguistics and Philosophy Alumni Office Information Systems Chemical Engineering Ocean Engineering Aeronautics and Astronautics Housing Mathematics Office of the President Medical Center Political Science ITAG Nuclear Science & Engineering Materials Science & Engineering Mechanical Engineering High Voltage Research Lab Agricultural Sciences Brain & Cognitive Science Ultimate Frisbee Club Toscanini's Biological Engineering Campus Police Property Office Technology Licensing Office Biology Economics The Tech Procurement Mechanical Engineering ... (plus a few hundred more)
An Application Project
An application programmer is charged to write an application to create Courses. This app starts the process by asking for a course code, title, and a department (the OSID calls this field the "sponsor" and happens to model it as a Resource).
The product owner wants the user to select from a list of known departments. The application programmer and the OSID implementation developer begin their collaboration. After learning about the Resource OSID, the application programmer adds this code:
public class DepartmentStuff {
private final org.osid.resource.ResourceManager resourceManager;
private static final String DEPARTMENT_RESOURCE_OSID_IMPL = "edu.school.orgs.OrgResourceManager";
public DepartmentStuff(org.osid.OsidRuntimeManager runtime) {
this.resourceManager = (org.osid.resource.ResourceManager) runtime.getOsidManager(org.osid.OSID.RESOURCE, DEPARTMENT_RESOURCE_OSID_IMPL, net.okapia.osid.kilimanjaro.OsidVersions.V3_0_0.getVersion()));
return;
}
public org.osid.resource.ResourceList getDepartmentsForCurriculum() {
return (this.resourceManager.getResourceLookupSession().getResources());
}
}
...and it produces this result:
Let the Iterations Begin!
This solution seems simple enough and the application programmer is pleased to show the product of his efforts. However, the product owner is not happy, Clearly there are many departments not relevant to the business of creating courses. The product owner asks to have this list pruned so only the applicable departments are displayed and sorted.
First, the application programmer sits down with the OSID implementation programmer to discuss the issue. They decide that Resources should be tagged somehow and stumble upon the genus Type that the OSID implementation programmer can supply and the application programmer can key off of. Looking up and down the hallway, they come up with the following types:
- administrative
- academic
- research
- student groups
- other
The OSID implementation programmer gleans this information by groveling through the Org hierarchies and other data in the data warehouse and writes the the logic to map it to one of the above Types. Now the application programmer can get to work:
public class DepartmentStuff {
private final org.osid.resource.ResourceManager resourceManager;
private static final String DEPARTMENT_RESOURCE_OSID_IMPL = "edu.school.orgs.OrgResourceManager";
public DepartmentStuff(org.osid.OsidRuntimeManager runtime) {
this.resourceManager = (org.osid.resource.ResourceManager) runtime.getOsidManager(org.osid.OSID.RESOURCE, DEPARTMENT_RESOURCE_OSID_IMPL, net.okapia.osid.kilimanjaro.OsidVersions.V3_0_0.getVersion()));
return;
}
public org.osid.resource.ResourceList getDepartmentsForCurriculum() {
java.util.List<org.osid.resource.Resource> orgs = new java.util.ArrayList<>()
try (org.osid.resource.ResourceList resources = getDepartmentResourceManager().getResourceLookupSession().getResourcesByGenusType(ACADEMIC_GENUS_TYPE)) {
while (resources.hasNext()) {
orgs.add(resources.getNextResource());
}
}
return (java.util.Collections.sort(orgs, new NameComparator()));
}
public static class NameComparator
implements Comparator<String> {
@Override
public int compare(org.osid.resource.Resource r1, org.osid.resource.Resource r2) {
return (r1.getDisplayName().getText().getCompareTo(r1.getDisplayName().getText()));
}
}
}
This is looking better and the OSID implementation programmer is starting to enjoy the agile groove.
The product owner sees Railroad Engineering... seriously? (That one hasn't been around since the 1930s.) The product owner explains that she wants "active" departments that are still around. With a touch of frustration, our application programmer sits down with the OSID implementation programmer once more. But what to do.
They decide that the issue is best solved by using a State to represent "active" (after all, the product owner used that word so it must have something to do with it). However, the data warehouse has effective dates in its Org table so they decide that the OSID resource Provider should examine these dates based on the current time to decide if they should be active or not.
After swearing at the OSID for not including a State inside the Resource, they look around and begrudgedly apply the Process OSID. Then they discover that an object might be in multiple states with respect to different Processes. They need to define a Process as well.
The application code becomes:
public class DepartmentStuff {
private final org.osid.resource.ResourceManager resourceManager;
private final org.osid.process.ProcessManager processManager;
private static final String DEPARTMENT_RESOURCE_OSID_IMPL = "edu.school.orgs.OrgResourceManager";
private static final String DEPARTMENT_PROCESS_OSID_IMPL = "edu.school.orgs.OrgProcessManager";
public DepartmentStuff(org.osid.OsidRuntimeManager runtime) {
this.resourceManager = (org.osid.resource.ResourceManager) runtime.getOsidManager(org.osid.OSID.RESOURCE, DEPARTMENT_RESOURCE_OSID_IMPL, net.okapia.osid.kilimanjaro.OsidVersions.V3_0_0.getVersion()));
this.processManager = (org.osid.process.ProcessManager) runtime.getOsidManager(org.osid.OSID.PROCESS, DEPARTMENT_PRROCESS_OSID_IMPL, net.okapia.osid.kilimanjaro.OsidVersions.V3_0_0.getVersion()));
return;
}
public java.util.Collection<org.osid.resource.Resource> getDepartmentsForCurriculum() {
java.util.List<org.osid.resource.Resource> orgs = new java.util.ArrayList<>()
try (org.osid.resource.ResourceList resources = getDepartmentResourceManager().getResourceLookupSession().getResourcesByGenusType(ACADEMIC_GENUS_TYPE)) {
while (resources.hasNext()) {
org.osid.resource.Resource resource = resources.getNextResource();
if (isActive(resource)) {
orgs.add(resource);
}
}
}
return (java.util.Collections.sort(orgs, new NameComparator()));
}
private boolean isActive(org.osid.resource.Resource resource) {
try {
org.osid.process.State state = getProcessService().getStateSession().getStateByReferenceAndProcess(resource.getId(), PROCESS_ID);
return (ACTIVE_STATE_ID.equals(state.getId()));
} catch (org.osid.NotFoundException nfe) {
return (false);
}
}
public static class NameComparator
implements Comparator<String> {
@Override
public int compare(org.osid.resource.Resource r1, org.osid.resource.Resource r2) {
return (r1.getDisplayName().getText().getCompareTo(r1.getDisplayName().getText()));
}
}
}
The product owner is pleased to see a complete and manageable list of subject areas and releases it to her customers.
The first feedback the project receives is that users are having a hard time finding these in the list. It turns out that at this institution the natural ordering is by course number, not name. The second feedback is to further filter the list to show only those departments to which the user is authorized to create courses. The application programmer negotiates with the product owner to forget the authorization thing and he'll figure out some way of re-sorting the data conditional upon continued cooperation from the OSID implementation programmer.
In the mean time, several departments disappear from the list. Upon investigation, the application programmer discovers that the Type of several Resources has changed from academic to administrative. The OSID implementation programmer explains that a financial application is also using this Resource OSID Provider and they were filtering out the "academic" and wanted to zoom in on "financial" Types. The Org hierarchy in the data warehouse had Orgs with multiple parents so he had fudged his code to bias toward "academic" since he didn't know what to do in this case but knew he could have only a single Type. The OSID implementor forgot about that when working with this new project where further tweaks had the side effect to change the bias and broke the course application.
While the OSID implementor struggles to solve this dilemma (how can I do it both ways at the same time?) he starts to yearn for the days application programmers were allowed to use SQL directly. The application programmer is also regretting this little endeavor and goes on to advise the product owner to set up their own database of organizations that can be controlled without having to negotiate the service contracts.
Intervention
In our not so made up example, we have three roles at this party. A product owner, an application programmer, and an OSID implementation programmer. While the product owner should understand the business and priorities well, a product owner may not be able to see what is happening architecturally. The application programmer knows how to build applications, and with each iteration cares less and less about what he's told about how things ought to work in an enterprise. The OSID implementation programmer has no visibility into the business while wandering into the role of infrastructure provider for all people. The development process is in trouble.
We're missing the architectural view. If the "architect" shows up with stack diagrams showing how AngularJS, Spring, Hibernate, and MySQL relate to each other, thank them and find a services architect who can slice through this problem before it gets worse.
Divide and Conquer
Coming in late to the party, the service architect has to deal with the desire to treat these services as a reusable infrastructural components but also has to be able to solve business-specific needs and relieve the mounting tension on the project. A lot of effort has already been expended to display a list of 22 departments that won't change in the lifetime of this project. The first thing the service architect does is whip together a simple Resource OSID Provider that stores the 22 Resources in a hashmap and spits them out in the right order. She instructs the application programmer to go back to his first version:
The product owner sees what she expects. The service architect will get back to this later but reducing the touch points is a good strategy for a user facing application.
Next, the service architect turns her attention to what was happening underneath.
Aside: Touchpoint and Agreement Analysis
Before we continue on to the work of the service architect, let's take a closer look at what was happening during the earlier iterations. Some terminology:
- touchpoint: Where an OSID Consumer invokes an operation in an OSID Provider. The more touchpoints, in other words, the more intricate the dance is between an OSID Consumer and an OSID Provider, the more difficult it is to intervene.
- agreements: Agreements are places where an OSID Consumer and an OSID Provider must agree in order to operate. Agreements may be in band or out of band. In band agreements are defined in the OSID Specification such as rendezvouing at a specific interface that an OSID Provider is expected to support. Out of band agreements are not defined in the OSID Specification such as Types and Ids the OSID Provider is expected to understand. The more agreements there are, the more difficult it is to intervene.
User-facing applications are difficult to create and the kind of OSID Consumers that should have a minimal set of touchpoints and agreements. OSID Adapters are simple to create and we can do all sorts of one-offs for special cases. OSID Adapters will be part of the solution to move the service assumptions out of the application into these more replaceable modules where we can afford more touchpoints and agreements.
| Iteration 1 Touchpoints | in band agreement | out of band agreement |
|---|---|---|
getOsidManager |  | |
>> DEPARTMENT_RESOURCE_OSID_IMPL | | |
| getResourceLookupSession | | |
| getResources | ||
| total | 2 | 1 |
| Iteration 2 Touchpoints | in band agreement | out of band agreement |
|---|---|---|
| getOsidManager | | |
>> DEPARTMENT_RESOURCE_OSID_IMPL | | |
| getResourceLookupSession | | |
| getResourcesByGenusType | ||
>> ACADEMIC_GENUS_TYPE | | |
| DisplayText comparison ignores format Types | | |
| total | 2 | 3 |
| Iteration 3 Touchpoints | in band agreement | out of band agreement |
|---|---|---|
| getOsidManager | | |
>> DEPARTMENT_RESOURCE_OSID_IMPL | | |
| getResourceLookupSession | | |
| getResourcesByGenusType | ||
>> ACADEMIC_GENUS_TYPE | | |
| getStateSession | | |
getStateByReferenceAndProcess | ||
| >> PROCESS_ID | | |
| >> ACTIVE_STATE_ID | | |
| DisplayText comparison ignores format Types | | |
| total | 3 | 5 |
Resources & Organizations
There is more to organization management than the Resource OSID delivers. The Resource OSID is used as an abstract reference in the Course because a Course sponsor may be an organization, an individual person, a Program, or all of the above. This abstract reference does not necessarily mean that it is the best OSID to manage the underlying data. Our trusty service architect is familiar with the service definitions. She begins building up from the org tables in the data warehouse by first selecting the OSID that makes the most sense and working out from there. The service architect divides this overall problem in two major pieces to separate what the course application needs from what is the best fit for the organizational data. Closing the gap will be of later concern.
The Personnel OSID defines Organizations. Unlike Resources, Organizations are both Temporal and Federateable. This means that the dates in which an Organization is effective and the multi-parented hierarchical relationships are designed into this service. The OSID implementation programmer is charged with implementing the OrganizationLookupSession and OrganizationHierarchySession of the Personnel OSID and will be able to do so without making any assumptions about active states or genus Types. The divide and conquer strategy is removing assumptions (business logic) from both the application and the underlying provider.
Closing the Gap
The application wants one OSID but the implementation is of another OSID. The integration job is to connect them.
However, if the service architect simply maps the Organization entity to the Resource entity, the issues of Type and State resurface. The service architect decides that it is the Personnel OSID Provider that will be general purpose and the Resource OSID Provider will act as a special case OSID Adapter and orients its design to serve the course application.
The service architect wants to keep the course application's touchpoints as simple as possible (but no simpler) by having it fetch all available Resources in the default Bin. That leaves Resource OSID Provider to filter out Organizations that are not applicable for the course application.
public class AcademicDepartmentFetchingResourceLookupSession
extends net.okapia.osid.jamocha.resource.spi.AbstractResourceLookupSession
implements org.osid.resource.ResourceLookupSession {
private final org.osid.personnel.OrganizationLookupSession orgLookupSession;
private final org.osid.personnel.OrganizationHierarchySession orgHierarchySession;
private static final org.osid.id.Id ROOT_ACADEMIC_NODE = new net.okapia.osid.primordium.id.BasicId("school.edu", "orgs", "731328");
public ResourceLookupSession(org.osid.personnel.OrganizationLookupSession orgLookupSession,
org.osid.personnel.OrganizationHierarchySession orgHierarchySession
org.osid.proxy.proxy proxy) {
this.orgLookupSession = orgLookupSession;
this.orgHierarchySession = orgHierarchySession;
setProxy(proxy);
return;
}
@OSID @Override
public org.osid.resource.ResourceList getResources() {
java.util.Collection<org.osid.resource.Resource> resources = new java.util.HashSet<>();
try (org.osid.personnel.OrganizationList orgs = this.orgLookupSession.getOrganizations()) {
while (orgs.hasNext()) {
org.osid.personnel.Organization org = orgs.getNextOrganization();
if (org.isEffective() && this.orgHierarchySession.isChildOfOrganization(org.getId(), ROOT_ACADEMIC_NODE)) {
resources.add(new Organization2ResourceAdapter(org)); // a class to wire the Ids, names, descriptions together
} // and set a Resource genus Type to "organization"
}
}
return (new net.okapia.osid.jamocha.resource.resource.ArrayResourceList(resources));
}
}
The service architect removed the assumptions to what an "applicable" organization is from both the application and the base Personnel OSID Provider. It also removed the assumption as to what is "active" (org.isEffective() should be only evaluated based on the current time, it will also be evaluated against the current "context" time).
This Resource OSID Provider is the place where this "business" logic resides and although it is not as reusable as the other components of this system (more touch points and agreements), it is also fairly lightweight.
Building Out
The service architect then learns of the feature the product owner wanted to narrow down the list of Orgs based on the authorizations of the user as a way of helping them streamline their workflow. A change is made:
if (org.isEffective() && this.orgHierarchySession.isChildOfOrganization(org.getId(), ROOT_ACADEMIC_NODE) &&
isAuthenticated() && this.authorizationSession.isAuthorized(getAuthenticatedAgentId(), CREATE_COURSE_FUNCTION_ID, org.getId()) {
resources.add(new Organization2ResourceAdapter(org));
}
As it turns out, there is an Authorization OSID Provider that uses a central authorization database that manages who can create courses under specific academic departments.
The application programmer is asked to make a change to pass in the "context" of the request via a Proxy so that the Resource OSID Provider has access to the authenticated Agent. The service architect had a Proxy OSID Provider that was able to do this from the servlet request that the programmer buried behind a makeProxy() ethos.
public class DepartmentStuff {
private final org.osid.resource.ResourceProxyManager resourceProxyManager;
private final org.osid.proxy.ProxyManager proxyManager;
private static final String DEPARTMENT_RESOURCE_OSID_IMPL = "edu.school.adapters.AdaptingOrgResourceManager";
private static final String DEPARTMENT_PROXY_OSID_IMPL = "edu.school.proxy.ServletProxyManager";
private static final org.osid.type.Type HTTP_SERVLET_REQUEST_PROXY_CONDITION_RECORD_TYPE = new net.okapia.osid.primordium.type.BasicType("edu.school", "proxy condition", "servlet request");
public DepartmentStuff(org.osid.OsidRuntimeManager runtime) {
this.resourceProxyManager = (org.osid.resource.ResourceManager) runtime.getOsidProxyManager(org.osid.OSID.RESOURCE, DEPARTMENT_RESOURCE_OSID_IMPL, net.okapia.osid.kilimanjaro.OsidVersions.V3_0_0.getVersion()));
this.proxyManager = (org.osid.proxy.ProxyManager) runtime.getOsidManager(org.osid.OSID.PROXY, DEPARTMENT_PROXY_OSID_IMPL, net.okapia.osid.kilimanjaro.OsidVersions.V3_0_0.getVersion()));
return;
}
public org.osid.resource.ResourceList getDepartmentsForCurriculum(HttpServletRequest request) {
return (this.resourceProxyManager.getResourceLookupSession(makeProxy(request)).getResources());
}
private org.osid.proxy.Proxy makeProxy(HttpServletRequest request) {
org.osid.proxy.ProxySession session = this.proxyManager.getProxySession();
org.osid.proxy.ProxyCondition condition = session.getProxyCondition();
edu.school.records.HttpServletRequestProxyConditionRecord record = (edu.school.records.HttpServletRequestProxyConditionRecord) condition.getConditionRecord(HTTP_SERVLET_REQUEST_PROXY_CONDITION_RECORD_TYPE);
record.setRequest(request);
return (session.getProxy(record));
}
}
The product owner has the UI changed such that if there is only one result, the user doesn't have to do anything. If there are no results, then there is an authorization misalignment in the Authorization OSID Provider(s) used among the Course OSID Provider and this Resource OSID Provider that should be addressed administratively.
The Loose Ends
Configurability of the OSID Providers
The application has four out of band agreements:
- on DEPARTMENT_RESOURCE_OSID_IMPL
- on DEPARTMENT_PROXY_OSID_IMPL
- that DEPARTMENT_RESOURCE_OSID_IMPL is compatible with DEPARTMENT_PROXY_OSID_IMPL
- that DEPARTMENT_PROXY_OSID_IMP supports HTTP_SERVLET_REQUEST_PROXY_CONDITION_RECORD_TYPE
The Orchestration OSID can be used to centralize these agreements with a single agreement on DEPARTMENT_ORCHESTRATION_OSID_IMPL and provides a means of changing the other OSID Providers without having to do so in the code.
public DepartmentStuff(org.osid.OsidRuntimeManager runtime) {
org.osid.OrchestrationManager orchestration = (org.osid.orchestration.OrchestrationManager) runtime.getOsidManager(org.osid.OSID.ORCHESTRATION, DEPARTMENT_ORCHESTRATION_OSID_IMPL, net.okapia.osid.kilimanjaro.OsidVersions.V3_0_0.getVersion()));
this.resourceProxyManager = orchestration.getResourceProxyManager();
this.proxyManager = orchestration.getProxyManager();
return;
}
If one wanted to go a step further, DEPARTMENT_ORCHESTRATION_OSID_IMPL could be pulled from the runtime configuration. It merely replaces the out of band agreement with the configuration Parameter Id but allows for a change of the OSID Orchestration Provider without having to recompile (assuming the Parameter Id is the more stable of the two).
Performance
While this system works, hitting the backend systems for each request leaves something to be desired.
The stars above show caching opportunities that could be implemented by an OSID Adapter:
- Cache the Organization results on a user by user basis.
- Cache all the Organizations stored in the data warehouse.
- Cache authorizations.
- on a user by user basis
- on a user by user basis for the specific Function and Qualifier
There's certainly no need to do all three. If (2) and (3) are implemented, then the evaluation loop in the Resource OSID will probably be fast enough.
There's another possibility depending on how the Personnel OSID is implemented. If it is capable of passing a query to the data warehouse, then it may also opt to support an OrganizationQuery. It may be in the interest of the Personnel OSID Provider to be used in an efficient manner. If the Personnel OSID Provider is inefficient, it gets cached behind its back. If it is fast and reliable, then it will likely get consumed as is. So, the extra work in supporting the OrganizationQuerySession may be valuable from the perspective of that provider.
Let's say that our OSID implementation programmer was already on the case and supported queries. This provides a different way for the Resource OSID Provider to do its thing.
public class AcademicDepartmentFetchingResourceLookupSession
extends net.okapia.osid.jamocha.resource.spi.AbstractResourceLookupSession
implements org.osid.resource.ResourceLookupSession {
private final org.osid.personnel.OrganizationQuerySession orgQuerySession;
private static final org.osid.id.Id ROOT_ACADEMIC_NODE = new net.okapia.osid.primordium.id.BasicId("school.edu", "orgs", "731328");
public ResourceLookupSession(org.osid.personnel.OrganizationQuerySession orgQuerySession,
org.osid.proxy.proxy proxy) {
this.orgQuerySession = orgQuerySession;
setProxy(proxy);
return;
}
@OSID @Override
public org.osid.resource.ResourceList getResources() {
org.osid.personnel.OrganizationQuery query = this.orgQuerySesssion.getOrganizationQuery();
query.matchEffective(true);
query.matchAncestorOrganizationId(ROOT_ACADEMIC_NODE);
try (org.osid.personnel.OrganizationList orgs = this.orgQuerySession.getOrganizationsByQuery(query)) {
while (orgs.hasNext()) {
org.osid.personnel.Organization org = orgs.getNextOrganization();
if (isAuthenticated() && this.authorizationSession.isAuthorized(getAuthenticatedAgentId(), CREATE_COURSE_FUNCTION_ID, org.getId()) {
resources.add(new Organization2ResourceAdapter(org));
}
}
}
return (new net.okapia.osid.jamocha.resource.resource.ArrayResourceList(resources));
}
}
Using the OrganizationQuerySession can increase direct performance between the Resource OSID and Personnel OSID. The touchpoints in the OrganizationHierarchySession were traded off for the extra touchpoints in the OrganizationQuerySession.
Retrospective
- It was difficult looking through this problem completely from the perspective of either the course application or the underlying system. Doing so pushed a burden to one side or the other. The piece in the middle is able to absorb the integration problems and simplify both the course application and underlying provider. In more complex scenarios, one-sided integration solutions (putting all the logic in the application or the underlying provider) becomes more fragile.
- Collaboration is considered a good thing yet the service architect inserted herself in the middle separating the two parties. This helped because each party could no longer reach over the line to react to the issues of the other. These issues, as in this case, are often integration related. These are better handled through a third-party integration solution that simplifies the overall problem.
- Within the service framework, getting it done now from a product owner perspective is a viable (and fruitful) option. The service architect initially supplied a cheap OSID Resource Provider that was simply echoed some static data. It wasn't what was desired in the end game, but it allowed the course application project to proceed while she worked on the other problems. The service architect did not block the application's progress while the other issues were being sorted.
- The integration issues can be considered "business logic" that did not belong at either endpoint. Business logic changes and we do not want it tangled up in our applications or system infrastructure. When the examination of the Organization hierarchy is no longer a viable way to determine the applicable departments, this "rule" can be changed accordingly without upsetting the code in either side because the service contract encapsulates it.
- The service architect did not concern herself with performance until it was all working. The performance can be experienced and analyzed through the lens of functioning code to see where the bottlenecks actually. Often these issues can be mitigated through the addition of new OSID Adapters as needed. It may also turn out that technological choices such as the backend protocols and databases cause performance issues. These can be identified and dealt with iteratively and with the appropriate people without upsetting anything else outside the affected service.
- The OSIDs are an architectural framework that can be applied in a variety of ways. As such, it's more about understanding tradeoffs than coming up with a single precise answer to every problem. Hopefully, this example scenario demonstrated the different ways this simple problem could be sliced and some solutions lend themselves to more successful iterative development than others.
- Iterative development is good when there's a solid architectural footing. Adapters can always be rearranged and replaced. Code can always become better tuned and more configurable. Iterative development without the architectural perspective risks creating a bigger knot when living in a world of changing requirements.