stressfree - methodology

The thesis aim: Industry applicability

David — Tue, 29 Aug 2006 00:16:00 +0000

The last couple of months have been difficult as I have tried to focus on the problem of satisfactory testing the Project Information Cloud concept. This process has required I nail down my aim and hypothesis so that whatever methodology was decided on would ensure the overall ambitions and specific requirements of the thesis were met. Initially my aim focused on improving the access, timeliness and relevance of the information available to project participants however this posed problems around the semantics and testability of ‘relevance’. Relevance can mean many things to many people and is also dependent on time. Something relevant to me now may not be relevant later but that same thing could be very relevant to you later on but of completely no value at the present. Consequently determining a reasonable and sound methodology for testing relevance was proving to be difficult.

The issue with testing relevance is establishing a viable control sample and locking down the constraints between the control and test samples to the point where the test becomes repeatable. Relevance is like hindsight, it is something you only know after the fact and as a consequence the relevance of data is usually dependent on a number of variables such as the person, situation, time constraints and the interface to the data. As a result I began to feel any test for relevance, be it through case studies, model based testing, prototyping or thought based experiments would end up testing these other factors more than the concept of the Project Information Cloud.

Mike and I talked about this a couple of Fridays ago during one of our weekly meetings. He raised a good point which was that one the seemingly forgotten origins of the research was that whilst the ‘Building Information Model’ approach was completely viable its timeframe for industry-wide adoption was very large. This long-term adoption process did not solve the industry’s short-medium term information management issues and if for some reason BIM failed to take hold as many felt it would AEC professionals would be left using the same basic collaboration and information management tools that they have been struggling with at the present time. Hence the primary underlying objective of the research was to propose and prove a method of improving the access and timeliness of project information that can be applied within todays current industry conditions.

Establishing ‘applicability’ as an aim instead of ‘relevance’ achieves two things. Firstly it grounds the research within the current AEC industry instead of setting it apart as some hypothetical and generalised project/information management thought experiment. This is important because after all this is an architecture thesis and should first and foremost be about an architecture problem not a series of general knowledge management concerns. Secondly by focusing on applicability rather than relevance it opens up a whole field of thought and prototype style experiments related to the Web and getting the concept to work in ‘practice’ (both in the real world and architecture sense). Following the applicability route allows the generation of far more specific and to the point conclusions, for example “this concept would work in contemporary practice given the following criteria are meet and these processes followed”.

Following this line of investigation also brings into focus the importance of not creating another Building Information Model concept in the sense that it should be immediately applicable and also not just a Web-enabled Building Information Model. Using this line of thought there are a number of Web-orientated conceptual and technical tests that can satisfactory test the immediate applicability of a Web-based, distributed Project Information Cloud. If it can be shown that the concept meets the requirements laid out by notable researchers such as Tim Berners Lee's Principles of Design then it can be argued that the concept would have a practical and immediate future within the Industry. Using this testing method it gets around the ever-present user-interface issues that exist when performing prototype or discussion group style experiments. In these scenarios often what ends up being tested is the quality of the human interface rather than the benefits or drawbacks of different practical solutions.

Useful PhD methodology links

David — Tue, 13 Dec 2005 01:49:23 +0000

I am stuck fleshing out my methodology, the idea is there but I have 'methodology block', probably more from aprehension than any real reason. I know I am going to create a prototype system for use in the BBSc303 course and perform some case studies of selected architecture practice projects. My main problem areas are how exactly will the prototype be tested and how will these case studies provide for generalisation of information needs rather than just highlighting unique events within a particular project. Consequently I have been trying to find useful stuff to help me out. A few things I have found so far are:

Writing and Presenting your Dissertation - S. Joseph Levine, Ph.D. (PDF Version)

Choosing your research methods wisely - Lea Winerman

An overview of qualitiative research methods

On a different tangent was this crazy site that appeared on the side of most of my Google searches. It is very crazy to think someone has made a business out of mass-producing papers for lazy students.

Meeting with Mike 13/9/05

David — Mon, 19 Sep 2005 23:53:06 +0000

Last Tuesday Mike and I met for a discussion on things. I was meaning to put an overview of what we talked about online sooner but it slipped my mind. Actually a far more interesting thing entered it - the home theatre system I bought the next day...

For the most part we discussed how the concept of 'rich and unobstructive' connections could be made a reality. Definitely the 'tagging' concept so well implemented in systems like Flickr would be a real benefit in such a system especially when it comes to the difficult task of categorising resources (text, images and CAD files) for searching. The ability for users to easy tag resources (be it theirs or others) would enable a degree of human searching not present in a contemporary model. Once use instance of this could be the client tagging product fittings they like and then having the architect place style tags next to them (i.e. postmodern, classic). Using these tags you could then pull slices out of the conceptual work such as 'classic, low-cost fittings that we (as in the client and architect) like'. Alternatively in a construction scenario the ability for the contractor to tag documents pertanent to a specific contractor would ease some documentation headaches (if I change this drawing who will be effected?).

A problem highlighted in Mike's thesis is the unreliability of hyperlinks as reference items over a long period of time. Whilst very reliable over a short period of time hyperlinks a prone to break over time with domain changes, site outages and corporate rebranding. Consequently the use of Internet based references for this thesis and arguably for the entire concept, should be backed up by some form of backup document to prove the initial hyperlinks existence. In the case of my thesis this will be achieved by saving Internet based resources as PDF. In the case of the sharing concept as a whole the problem is a little more complex and I will go into that a little later. Mike brought up the existence of the Wayback Machine. Not only do they run this archive on some fairly serious hardware (http://www.archive.org/web/hardware.php) but they also keep a fairly comprehensive archive of the web going back over seven years. I have just entered my site into the WayBack machine's web crawler so hopefully in the future copies of this site will be around for posterity and to cringe at.

Our last topic of discussion was how this concept could be tested. The major problem with a quality test (is my thing better than the conventional process?) is that it is almost impossible to test or prove conclusively. In theory a quality test such as this could be run on hundreds of real world projects and the results, in the form of questionnaires and evaluation forms, compared and summarized. Unfortunately such a test is not practical and anything similar performed on a small scale would not provide conclusive evidence.

A case study approach could be a better strategy as it substitutes depth for breath. In this form of testing a few projects could be evaluated extensively using interviews, discussion groups and observation to determine the impact (if any) of the proposed system. In order to prove the feasibility of the system the scalability of the concept would need to be proven. To prove this the system could be tested in mockup small and large practice environments and testing how it performs under a variety of conditions. The common belief is that something that can be 'scaled up' is successful but I think the reverse principle should be used for this test. I have found in practice that anything can be 'scaled up' given enough investment in resources, support and training. The real test of a system is how it can 'scale down' into low resource environments such as that of a New Zealand architecture practice and then be scaled back up into a full scale enterprise environment (100+ users from many different companies participating).

The other test of how successful the concept will be is in how robustly it performs during failure of part of the network. This idea touches on the problems Mike has experienced with Internet based references no longer being accessible on the Internet for one reason or another. In practice this same problem will be faced by users of the system, especially in environments where multiple organisations are sharing information. In this scenario it would not be unreasonable to suggest that at least once during the development process one of the companies would temporarily (or permanently) drop out of the information network for technical or business reasons. When this circumstance arises the system must still be able to provide the information submitted by this party else the integrity of the system as a whole would be severely compromised. No user wants to be told to 'come back tomorrow' if time critical information is not available when they need it and likewise a project's information should not be held to ransom by the threat of one business failing to live up to their service agreement. Consequently the system must use caching not only to ensure robustness of service but also to aid in boosting the overall user experience through speedy service, especially when large data files are involved. There is an interesting speech by Justin Chapweske of Onion Networks available at IT Conversations that deals with this subject (http://www.itconversations.com/shows/detail462.html).
The complete Onion Networks SDK costs US$25,000 which puts it out of reach for this application but definitely the concepts (and the Public Edition) are very useful to keep in mind.