HT/VT Web Site

48 postsstarted by WelchTC 16,897 viewsoriginal topic

Senior Member
#1
The [thread=502]HT/VT Website Overview thread[/thread] has started some discussions at the Church about how could utilize the community to build this type of application officially for the Church. Here are some of the challenges that we face and we would appreciate any feedback you as the community could give us in order to decide if this is a project that we can properly control and outsource to the community.
  1. How do we securely integrate this into internal systems here at the Church that are currently under development without exposing sensitive and critical data to the community. If we provide API's or other Web services to integrate into these systems, do we expose the Church and the privacy of the data to people wanting to use those API's for non-approved purposes?
  2. How do we maintain quality of the code that is being submitted.
  3. How do we prevent code from being submitted and accepted that has "back doors" into confidential data.
  4. How do we best "beta test" such software?
  5. Should we open this application up to anyone who would like to contribute?
  6. What roles should the community play? (Development, Documentation, User Interaction, QA, Project Management, others I'm missing). How would we handle these roles. It is easy to say "all of these" but how would we do that? How do we have a community project manage the app, for example?
  7. What should we be concerned with that I've not mentioned?
Many features have already been discussed. I'd like to post some of the basic objectives we would need to accomplish and open this thread up to other objectives and features not previously mentioned in the other thread. Remember, we are in exploratory mode. We are not committing to open this project up to the community but are simply exploring if it is feasible as well as getting great ideas from the community.
  1. Assign responsibility to priesthood bearers and sisters for particular individuals and families.
  2. Report on activities (visiting, calling, writing...) by those assigned responsibility to watch over another.
  3. Make confidential comments visible only to Bishops/Branch Presidents.
  4. Produce reports including: who is assigned to who, who is not assigned a responsibility to watch over another, who has no one assigned to watch over them, summary of activities by the assigned and by those watched over.
  5. All this must be available via the web.
Let us know what you think.

Tom
Community Moderators
#2
I think it's a great idea to be considering outsourcing something like this to the community. You asked for thoughts/ideas, so here are a few:

To be able to securely integrate this with internal systems, the security aspects of the high-level design should probably be done in-house, with reviews as appropriate to ensure adequate protection mechanisms are built in from the foundation up.

For code quality and assurance against back doors and such, one approach would be to assign a small number of Church-employed developers to be part of the team, to be involved in and to some extent oversee design, coding, reviews, etc. Code review by a combination of internal Church employees and the rest of the community team would seem to be a good approach for both effectiveness and efficiency.

On whether to open the project up to everyone, perhaps one model to consider would be something like what is used for early morming seminary teachers. An interview by the prospective developers' eccliastical leaders would provide a fairly good screen against someone who would want to insert back doors or other problems. At least when I taught early morning seminary, CES paid the teachers a small stipend, reportedly mostly for the purpose of enabling CES to fire the teacher if they wanted to.

Another thing to look into is whether the employers of volunteer developers claim copyright on anything the developer writes. The employement agreement with my former employer included a clause that gave the employer ownership of the copyright on anything the employee wrote, even off hours, developed entirely without any use of company resources, etc., for anything that was related to the company's business. I understand some open source processes require developers to affirm that their employer disclaims ownership of copyright on what the developer will be submitting.
Community Administrator
#3
One thought might be to have the church do the API and the community do the UI. The UI never gets access to data that the user isn't permitted to see. That would solve a number of problems and lifts a fair-sized burden from the church developers.

For testing, MLS has a demo ward/stake. Perhaps that can be extended to LUWS?
Have you searched the Help Center? Try doing a Google search and adding "site:churchofjesuschrist.org/help" to the search criteria.

So we can better help you, please edit your Profile to include your general location.
Senior Member
#4
Great comments...keep them coming.

Tom
Church Employee
#5

LDS Account?

I would assume that using the new LDS Account to authenticate / authorize users for a specific church unit would help alleviate much of the security concerns, as long as the secure data access is done via web services which validate that the user has logged in. The only data which you could retrieve would be for the unit that you already have access to via LUWS. (So yes, there is still a privacy concern, but not significantly more than what has already been exposed)

If this assumption is valid, then other non-employee developers can still work on the site (except for those portions which requires direct access to the full data).
Church Employee
#6

code reviews, code quality

As far as verifying the quality of the code and ensuring that no security holes are introduced... I think that we can follow the model that many other open source applications have utilized. As long as enough people are actively working on the project, you will be able to quickly identify bugs and keep the quality up.

You would only have a few trusted individuals who are "committers" on the project. It is their responsibility to review any code submissions, ensure standards are met, and test are written before their changes are merged into the main part of the code base... otherwise the changes are rejected. This ensures that only small, incremental changes are made by "unknown" contributors and larger, global changes can really only be made by committers. Individuals who gain a reputation for good, quality code can be upgraded to a committer role, while those who have questionable contributions can be demoted so that all submissions are scrutinized more closely.
Community Administrator
#7
As for testing - expanding on my idea of the test ward/stake, how about test membership record numbers and test logins to LUWS? The logins can be shared among testers allowing multiple people to test the different roles. Done right I think we can run the test on a live system without messing up the real data. We just have to be careful about move-in and move-outs that they don't go to real units.

Then again, how does the church test their back-ends? Do they have a membership system testbed?

Oh, and those test units: Ward Unit #108, Stake Unit #2224445.
Have you searched the Help Center? Try doing a Google search and adding "site:churchofjesuschrist.org/help" to the search criteria.

So we can better help you, please edit your Profile to include your general location.
Member
#8
What about creating test Web Services that are somehow made available only to developers via the Internet (perhaps anyone that has an account with a custom access level on this message board?) The Web Services contain fake test data, but are otherwise identical to the Web Services that are used internally by Church systems. Perhaps a website could be created that displayed committed data (for testing purposes).

This way, applications could be developed and tested outside of the Church infrastructure, but would, in theory, function properly once they were brought internally and pointed to the internal Web Services.

Brad O.
Senior Member
#9
I have an idea or two as well.

There could be three open test periods once the development is close to release.

1. Final alpha: At Church headquarters, using Church employees and department volunteers.

2. First beta: Outside (this community) developers.

3. Second beta: Outside: This community and actual priesthood leaders and others who might wish to communicate with computer skills to simulate what happens when new people are called and have to interact with the system, typically for the first time or after some time with new updates. An update to the code should be done during the beta to similate how to debug on the fly for actual users who encounter problems using the system.

In all three cases several things would be happening: The test units would have move-ins and move-outs and leadership changes and that last one would be accelerated more than you would have in a normal ward or stake. Maybe people in the community could play the role of the 'leaders' that are called and released, as some of those here like myself are serving in various leadership positions that relate to all of this. Even that could help make the final product ready for prime time.

Any thoughts?.
Senior Member
#10
By the way, Tom, this is really exciting news!
tomw wrote:How do we securely integrate this into internal systems here at the Church that are currently under development without exposing sensitive and critical data to the community. If we provide API's or other Web services to integrate into these systems, do we expose the Church and the privacy of the data to people wanting to use those API's for non-approved purposes?
I understand what you're asking, but yet I don't think I am grasping all of the concerns...Are you speaking about security vunerabilities of the data itself, exposure of internal Church systems, or maybe both? If it's the data, what type of data might be passing through the app of a secure nature? If Church systems, are we talking solely about databases, or does it go further than that? If I can get a better idea of the concerns here, I might have some decent comments to contribute...Thanks.
tomw wrote:What roles should the community play? (Development, Documentation, User Interaction, QA, Project Management, others I'm missing). How would we handle these roles. It is easy to say "all of these" but how would we do that? How do we have a community project manage the app, for example?
Ya know, I think "all of these" might be the right answer though (perhaps with the exception of "Project Management," that might be best left to Church employees). But I really do think we have a diverse enough crowd here to merit such diversity in community roles. Everyone here knows there strengths, time constraints, preferences, etc. There are some here that just love to develop no matter what it is; others may love to try and find the bugs in the system and even have access to automated tools to facilitate QA; I have even seen a few posts on the forum of those who love to help improve the documentation. Perhaps we open the choice of roles to the user, allowing each volunteer to take on up to 2 or 3 roles. Another option would be to have some questionare describing skill sets and experiences, and allow Church employees to assign roles as deemed appropriate (I have a feeling it might get weighted heavily toward the "Developer" role, and it may take some encouraging from Church employees to get some to take on some other roles).
One thing I would like to say is that I really appreciate what the newly created User Interaction team at the Church has done with the updated look-and-feel of all Church sites. I think some cooperation and/or guidance from that team would be terrific. I really like what brado426 has done so far, and I think it would be neat to have it fit right in with the look of other Church sites.
Member
#11
mkmurray wrote:I really like what brado426 has done so far, and I think it would be neat to have it fit right in with the look of other Church sites.

Thanks, mcmurray. The HT/VT Website is written in C# utilizing Microsoft ASP.NET 2.0. It also uses cascading style sheets for the look and feel. I have not asked about or looked into what the Church uses to style their websites, but I'm guessing they probably have a library of CSS styles somewhere. I haven't really bothered styling this HT/VT application yet because I figured the Church probably had a library of their own that would make the process a lot easier. I'm no artist... I can make a site functional, but I'm not so great at making it look pretty; So any shortcuts I can take to make my websites look half-way decent are welcomed. :)

It would be nice if such a stylesheet library was provided to the community.

Brad O.
Member
#12
tomw wrote:
  1. How do we securely integrate this into internal systems here at the Church that are currently under development without exposing sensitive and critical data to the community. If we provide API's or other Web services to integrate into these systems, do we expose the Church and the privacy of the data to people wanting to use those API's for non-approved purposes?


The API's/web services would not have to have direct access to sensitive data if they only work server-side (e.g. php). Developers could work with a local mySQL/php mockup of the website with a test API designed for that, then upload their changes to the code repository for approval. Only approved changes would be added to the site and have access to the 'real' API and data.

One side effect of this, which has both positive and negative aspects, is that people would not be able to create their own live programs for using the API (e.g. create a website on their own site that ties in to LUWS that farms for emails/phone #s to advertise some lds-oriented product/service - I can definitely see this happening semi-innocently). The only projects with access to live info would be those approved by the development team.
Member
#13
haledn wrote:As far as verifying the quality of the code and ensuring that no security holes are introduced... I think that we can follow the model that many other open source applications have utilized. As long as enough people are actively working on the project, you will be able to quickly identify bugs and keep the quality up.
I agree with haledn. We should be successful following the project model for successful open source projects. They've proven to be high quality, efficient, and secure.
Senior Member
#14
RussellHltn wrote:As for testing - expanding on my idea of the test ward/stake, how about test membership record numbers and test logins to LUWS? The logins can be shared among testers allowing multiple people to test the different roles. Done right I think we can run the test on a live system without messing up the real data. We just have to be careful about move-in and move-outs that they don't go to real units.

Then again, how does the church test their back-ends? Do they have a membership system testbed?

Oh, and those test units: Ward Unit #108, Stake Unit #2224445.
We do have test data and I would suppose that we would setup a test data environment.

Tom
Senior Member
#15
mkmurray wrote:By the way, Tom, this is really exciting news!
I understand what you're asking, but yet I don't think I am grasping all of the concerns...Are you speaking about security vunerabilities of the data itself, exposure of internal Church systems, or maybe both? If it's the data, what type of data might be passing through the app of a secure nature? If Church systems, are we talking solely about databases, or does it go further than that? If I can get a better idea of the concerns here, I might have some decent comments to contribute...Thanks.
I'm not sure if I have any concrete examples or explainations. I'm curious as to what the community feels are our security concerns. From the Church's position, there is a lot of very sensitive data that we need to all access to only those who should have it. Obvioiusly we can create a test database of data that developers can hit but once a developer knows how things work, are we exposed to developer hacks to get around security?

Tom[/align]
Community Moderators
#16
HPaulsen wrote:[/align]The API's/web services would not have to have direct access to sensitive data if they only work server-side (e.g. php). Developers could work with a local mySQL/php mockup of the website with a test API designed for that, then upload their changes to the code repository for approval. Only approved changes would be added to the site and have access to the 'real' API and data.

One side effect of this, which has both positive and negative aspects, is that people would not be able to create their own live programs for using the API (e.g. create a website on their own site that ties in to LUWS that farms for emails/phone #s to advertise some lds-oriented product/service - I can definitely see this happening semi-innocently). The only projects with access to live info would be those approved by the development team.

Though since they would have to use their own username and password they would be limited to the data they have access to and so they could only use it as they can now. If someone really wanted to harvest the data just download the vCard of the stake directory and parse it. Not much harder then using the API. Now if you are taking about global information on the church that I believe is a NO NO for anyone. And of course you could only change your personal information. So I think the API should be able to read what your username has access to and change what your username can normally change.
- David
Community Moderators
#17
tomw wrote:
I'm not sure if I have any concrete examples or explainations. I'm curious as to what the community feels are our security concerns. From the Church's position, there is a lot of very sensitive data that we need to all access to only those who should have it. Obvioiusly we can create a test database of data that developers can hit but once a developer knows how things work, are we exposed to developer hacks to get around security?

Tom

As I see it if the API requires you to securely initiate a session with the server using your username and password then you'd be limited to what your username can do on the site anyway. I just see the API as a way to interface with LUWS or any LDS service with slim data that can be used quickly in the program.

As it is right now you have to parse the webpages to get access to the data and that can be annoying since we all know that HTML is not very strict and isn't a good method for transmitting data (thus XML was created).
- David
Community Moderators
#18
bhofmann wrote:I agree with haledn. We should be successful following the project model for successful open source projects. They've proven to be high quality, efficient, and secure.

Also open-source with enough developers means that projects won't stagnate.
- David
Community Administrator
#19
tomw wrote:Obvioiusly we can create a test database of data that developers can hit but once a developer knows how things work, are we exposed to developer hacks to get around security?
I think there is some small exposure. But then I think you are more likely to have someone point out the flaw.

Which is more secure - security though obscurity, or security though a bunch of computer geeks looking over the shoulder - all with a vested interest in keeping things secure? I will admit the first case is more smooth and peaceful. :D
Have you searched the Help Center? Try doing a Google search and adding "site:churchofjesuschrist.org/help" to the search criteria.

So we can better help you, please edit your Profile to include your general location.
Member
#20
RussellHltn wrote:I think there is some small exposure. But then I think you are more likely to have someone point out the flaw.

Which is more secure - security though obscurity, or security though a bunch of computer geeks looking over the shoulder - all with a vested interest in keeping things secure? I will admit the first case is more smooth and peaceful. :D
I agree and we've seen this on this forum. When ever there is an infraction it is the general forum members that find and report more often than the moderators. I think it would be pretty difficult to slip something in with everyone collaborating on the project.
Member
#21
thedqs wrote:Also open-source with enough developers means that projects won't stagnate.
Just for clarity sake, it should be plainly stated that this would not be an open source project. The code would be developed by volunteers and owned by the church. The volunteers would not have liberty to use the code for any other purposes.

That said and understood the open source model would provide the security and conveniences this type of project would need.

Senior Member
#22
nimebe wrote:Just for clarity sake, it should be plainly stated that this would not be an open source project. The code would be developed by volunteers and owned by the church. The volunteers would not have liberty to use the code for any other purposes.

That said and understood the open source model would provide the security and conveniences this type of project would need.
We have not made any decisions (should we proceed with this project) if we would release the code to the community or not. Personally I can't see much value of having it "open source" as I don't know who would make derivative works from it.

Tom
Member
#23
Tom:

I'm not sure if this justifies a new post or is along the lines of what you are looking for in this thread. A comment you made in another thread brings up a point.

A community member working on an outsourced project for the church would need an adequate test environment at their home. Therefore, some instructions or possibly even tools would need to be made available to these community developers.

The thing that jumped out at me is when you mentioned that the church mostly uses Oracle for a back-end database. I have used Oracle at work for a couple development projects, but we mostly use MS SQL. I can get an MS SQL server set up easily because I have done it many times and Microsoft provides a free watered-down version to developers. Though I have done some development with Oracle, I have never set up an Oracle server... nor do I know whether a free version that matches the same version that the church uses is even available to run at home for development purposes.

In the case of the HT/VT Reporting site that I have been working on, it could be made to work with a Web Service to acquire and submit church data, but even then it would still require a back-end database of its own.

The point I am trying to make is that community members would need to be provided with clear documentation that describes how a test environment must be configured including DB server version/Java Version/Microsoft .NET Framework version, etc. Also, the community members would need to somehow have access to all the tools and development licenses that they need.

Brad O.
New Member
#24
The community members wouldn't need a DB server running on their own computer. The church would just need to create secure dynamic XML pages (example http://www.lds.org/data.php?key=k234ksd ... t=06234522) which require a specific key to access them much like the Google Maps API. Community developers could use AJAX to work with the content thus the data would stay secure and every one would have an even playing field for testing and developing.


Rich
New Member
#25
marlattrj wrote:... XML pages (example http://www.lds.org/data.php?key=k234ksd ... t=06234522) which require a specific key to access ...
FYI this is just a fake URL I created as an example. -Rich
Member
#26

Realistically...

Nothing finds flaws in applications faster than saying, "This application is secure."

You could release a test db and access calls and just require a week of community testing before any applications could be developed. Then use the honor system.
Mr. M
-----------------------------------------------------------------------
Visit me virtually anytime. ;)

http://www.mariohipol.com
New Member
#27
Mr. M wrote:Nothing finds flaws in applications faster than saying, "This application is secure."

I absolutely agree with you.
I was just suggesting that keys could be assigned to community developers, and that they could allow access to an API that outputs XML pages from the church DB. The nice thing about that would be that the church could control what they wanted to provide as public info and developers could use that data for development. XML is very nice to work with, a developer can make calls to a dynamic XML page using AJAX and create very nice applications.
Member
#28
I was just suggesting that keys could be assigned to community developers, and that they could allow access to an API that outputs XML pages from the church DB. The nice thing about that would be that the church could control what they wanted to provide as public info and developers could use that data for development. XML is very nice to work with, a developer can make calls to a dynamic XML page using AJAX and create very nice applications.

That sounds a lot like a .NET Web Service. I think that is a good way to exchange data between one system and another (for example accessing information in church systems), but I'm not so sure it is a good idea for an application to be designed to function solely using this technology.

I am wondering how hard it would be for the developer to modify the back-end database schema when doing it this way? Also, wouldn't there would be significant overhead running an entire application based on XML data feeds. Additionally, as a developer, I prefer to have full access to the application's database that I am working on... otherwise I think things would take much longer.

Perhaps I'm only apprehensive about it only because I've never done it before and I do not know of anyone that is doing it. :)

Brad O.
New Member
#29
brado426 wrote:That sounds a lot like a .NET Web Service. I think that is a good way to exchange data between one system and another (for example accessing information in church systems), but for some reason I'm not so sure it is a good idea for an application to be designed to function solely using this technology.
AJAX is not just a .NET method, in fact you don't need .NET at all. AJAX stands for Asynchronous JavaScript and XML, it is a method that allows websites to transfer data back and forth without requiring the page to be refreshed. which can greatly speed up web application as it you no longer have to reload an entire page. This is done using the XMLHttpRequest object in javascript sending and receiving data asynchronously. XML is the typical type of data that is exchanged although any format will work, including preformatted HTML, plain text, JSON, etc. These files can be created dynamically by some form of server-side scripting.

Most web apps are moving this direction (ie, Yahoo, Google, etc) due to the speed and and flexibility. I would think that the church would also want to start going in this direction as well.
brado426 wrote: I am wondering how hard it would be for the developer to modify the back-end database schema when doing it this way? Also, wouldn't there would be significant overhead running an entire application based on XML data feeds. Additionally, as a developer, I prefer to have full access to the application's database that I am working on... otherwise I think things would take much longer.
At first developers would need to work with the church to developed DB schema, but remember the data can be retrieved using a dynamic server-side script. so for instance the web page can call for a specific unit by passing the unit number and then pass other criteria to the server, the server would then send back a formated XML document with that units data for the page to read. I would suggest looking into this it is very slick.

I suggested this as a posible answer to Tom's first question, "How do we securely integrate this into internal systems here at the Church that are currently under development without exposing sensitive and critical data to the community....", this would give the church control over what data they allow to be accessed by the community.
Member
#30

AJAX is not just a .NET method, in fact you don't need .NET at all. AJAX stands for Asynchronous JavaScript and XML.

I was just comparing it to a .NET WebService.... I didn't mean that I thought .NET was required. I know that AJAX libraries have only been recently made available for .NET.

I think I understand what you are suggesting. I don't think I have enough experience with it to comment further though.

Brad O.
Senior Member
#31
brado426 wrote:Additionally, as a developer, I prefer to have full access to the application's database that I am working on... otherwise I think things would take much longer.
It's more for maintaing complexity and introducing modularity. You want to be able to "plug-and-play" the different parts of your application, in case changes in implementation have to be made. This is called "decoupling" the various parts of the application. When you get to complex applications, you typically don't want your Web Tier interacting directly with your Data Tier. There is usually some sort of abstraction in between, a Service Tier. This would allow you change the database being used to store the data, without having a direct influence on your web interface.

Especially in this circumstance where the Church would want to keep the database schemas and connection strings secure, I would think that would be another argument for not going straight at the tables in the database.
Community Moderators
#32
I have to agree with Mike on this one. If you start crossing levels then you have to maintain a lot more code for simple changes. Example the church could have a method getHeadOfHousehold(int FamilyID) which just searches through database and gets the person with PositionInFamily = 1 AND FamilyID = userinputFamilyID.
Now if they change the databases to have a HeadOfHousehold as a boolean value the getHeadOfHousehold method would change BUT the external program stays the same. So really having the database give no advantage and in my opinion gives actually a disadvantage to the programmer.
Give me a standard API and an secure socket connection to it and I'm happy. :D
- David
Member
#33
thedqs: I know what you're saying, I always do as you say and put database calls in their own methods so the database connector can be changed to whatever is necessary. As long as these methods return the datasets in the correct format, everything should work.

I guess what kind of bothers me is that it seems inefficient to be transmitting all that XML back and forth between the application server and the data server. Perhaps that is an old-school concern with today's abundance of processor, memory, and bandwidth.

Brad O.
Community Moderators
#34
brado426 wrote:I guess what kind of bothers me is that it seems inefficient to be transmitting all that XML back and forth between the application server and the data server. Perhaps that is an old-school concern with today's abundance of processor, memory, and bandwidth.

Even then I think we still need to be sparse but what is larger: An entire database of church information (cut down to what the user could use), or a short response to a query?

What I mean is though you may reduce total traffic, the speed of a 56 Kbit modem could handel the speed needed to transmit the XML formatted data that a human interacting through a program could use in real time. For example (might not apply to this thread but the idea is the same):
Someone has written a program that interfaces with LUWS ward list and displays photos if avaliable. (The photo part might take longer to do since over 56 Kbits that can bottleneck)
The program connets to the API interface (less then 5 KB of data or less then a second for connection time)
Program sends an XML formatted command to the API to get a list of ward members and it gets back an XML formatted list of members with phone address and picture IDs (I'll be generous and say that was 15 KB of data or abour 2 seconds). (The program then starts to download each picture by sending a XML request with the picture ID (This could end up in the MBs but since it is being run in the background as another thread the user only notices that the pictures take time to show and not any interference with the program)
The person wants to look at a detailed view of the family (maybe children with ages and personal email address and cell phone numbers) they click the family and the program sends a request for that family ID to get an XML table of the information of that family (5KB or less then a sec)). (The photo thread is still in the background getting pictures so there might be a slight delay for one image to complete and to get this request through but some good multi-threaded code should be able to make this effect disappear)
The user got his information and then closes the program and to the user it seems that the program was very fast.

Now if you download the entire database first (let's say pictures are store in the database too) you will have to download a multi-MB file before you could even start and the user has to wait a long time.

Hopefully this example shows why I think getting the entire database is not only wasteful but a bad idea personally.
- David
Senior Member
#35

Web Services

At the Church we don't even let the developers mess around with the database. We really try to force all code to go through a middle tier to ensure security and maintainability. So just because I said we use Oracle, does not mean that you would need to access Oracle directly. We (the Church) would develop the necessary API's to touch the database. It may seem inefficient but actually Web services offer a lot of advantages.

If this project gets off of the ground, we would work with the community to provide them with the necessary tools to do all of their development with. This would include web services to a "dummy database" that could be used for testing, etc.

Tom
Member
#36
Interesting. I guess one needs to be pretty sure that their database schema is accurate before having the church set up the middle-tier web-service you are referring to. No matter how much I plan, I always end up changing at least a few things in the database schema during development.

Is there any way to give the developer the ability to change the schema on the middle-tier so that they would not need to bother someone at the church every time? My concern is innundating church resources with requests during development. I'm sure an application could be written to allow this, but it sounds like a big job just to write that application. What process do developers at the church use now to get database schema changes made?

Brad
Community Moderators
#37
I am sure that the church wouldn't mind getting the requests basically since everyone probably wants the same functionality and the church most likely wants to review each request for immediacy and necessity. Also I am sure the church wants to keep a tight hold on the API tier.
- David
Member
#38
Sorry to have caused all this "back and forth" over this issue. Handling the back-end of the application this way is a new concept for me. However, I think you guys have me convinced that it is the best way to go... at least for the Church.

Brad O.
Senior Member
#39
brado426 wrote:What process do developers at the church use now to get database schema changes made?

Brad
During initial development many programmers may have access to a database while the DB schema is in flux. Once, however, it is fairly solid, the engineers work with a DBA to make the necessary changes. Yes, it is slower and inconvenient but it does protect the Church from one developer making an inadvertent change that messes up a whole bunch of other apps depending upon the same data.

Tom
Member
#40
I don't mean to open up another can of worms here, but I thought I should post some feedback I received after doing some additional research. Your comments have intrigued me quite a bit and I have asked a few people that I know about the proper use of WebServices that I consider more experienced than I in applications development.

I have compiled all of their comments into a few bullet-points:
-
  • WebServices should only be used for exchanging data between two systems that would otherwise be incompatible
  • Using WebServices require parse/deparse time
  • WebServices are more likely affected by network latency
  • WebServices include HTTP overhead
  • To keep developers from affecting the database schema, have the DBA lock it down
  • Use an API that provides "direct-access" to the database
-
I guess it is fine to use a WebService as the sole method of database interaction, as long as the trade-offs are fully understood.

Brad O.
Member
#41
I considered web services for a product I have been working on, but after doing some research came to the same conclusions - the network overhead was too great, particularly for the small data packets I intended to use. There are alternatives. Even just compressing the xml data can help on network traffic, though it adds even more processing requirements. For my program, this was fine because much of it would be done on the customer's machine. There are always trade-offs, and what method you use really depends on your priorities. If security and ease-of-use is desires, web services may be ideal. If processing time and/or bandwidth are concerns, then I'd consider an alternative.
Senior Member
#42
I agree with HPaulsen. The decision to use WebServices should be weighed by your priorities. I think everyone would agree that WebServices entail some overhead; although, sometimes the overhead is introduced because of the gain you get in abstraction, usability, and security.

Large, complex systems ideally should be designed to be modular and flexible (remember, that's what we shoot for, and not always the reality for the current implementation at some random corporation). I personally think this implementation will not be a one-time deal, only used for a HT/VT reporting module. If designed right, this system could be molded into many different uses and possibly display or store the data in many different forms, flexible enough for whatever the Church's needs are.

One thing to consider too is where this HT/VT Reporting module is to be hosted. Is it apart of MLS? LUWS? Is it hosted centrally at the Church for all units? Is it more of a client that has to make remote calls to where the data is? It may be the case that the module is not being hosted right there beside the data. In that case, WebServices might be the only option.

Personally, I think as complex and large as the Church's systems are and also as large as their requirements are, the WebService approach may have a big advantage in the long run.
Senior Member
#43

Web Services...

Web services are a must at the Church. You need to remember that not all data is stored in 1 big database but may be spread out across multiple systems. When talking about the HT/VT system, it will need to interface with the membership database(s) but could also integrate into other systems. Web services will make all of the integration into various systems transparent. Otherwise every developer on the team would have to know and understand each DB schema, rules, etc.

Web services also can provide much more security. Since membership data has to be kept extremely confidential, it is a must to provide as much security around this as possible.

Tom
Member
#44
Just to clarify, I wasn't trying to imply that I thought the Church was doing it the wrong way. I'm sure the Church has already done the research and selected the best method based on the pros and cons.

I just think we should be careful to be sure to state the pros and cons of using any technology here so that other technical people reading this can better make an educated judgement for themselves.

I was researching this because I was thinking about suggesting this method for my organization. However, after doing some additional research, I don't think it would work in our particular circumstance.

Brad O.
Senior Member
#45
brado426 wrote:Just to clarify, I wasn't trying to imply that I thought the Church was doing it the wrong way. I'm sure the Church has already done the research and selected the best method based on the pros and cons.

I just think we should be careful to be sure to state the pros and cons of using any technology here so that other technical people reading this can better make an educated judgement for themselves.

I was researching this because I was thinking about suggesting this method for my organization. However, after doing some additional research, I don't think it would work in our particular circumstance.

Brad O.
Thanks for the clarification.

Tom
Member
#46
brado426 wrote:A community member working on an outsourced project for the church would need an adequate test environment at their home. Therefore, some instructions or possibly even tools would need to be made available to these community developers.

The thing that jumped out at me is when you mentioned that the church mostly uses Oracle for a back-end database. I have used Oracle at work for a couple development projects, but we mostly use MS SQL. I can get an MS SQL server set up easily because I have done it many times and Microsoft provides a free watered-down version to developers. Though I have done some development with Oracle, I have never set up an Oracle server... nor do I know whether a free version that matches the same version that the church uses is even available to run at home for development purposes.
Most open source projects use open source tools that are available to everyone (like MySQL). I think this is why most open source projects are successful in a community environment.

Depending on the complexity of the project you might be able to export table schemas and sample data that can be executed in a variety of databases. Then developers can have a development environment with the database of their choice. Proper application architecture requires that data access be a separate layer anyway. Of course there are slight differences in SQL syntax between some databases...

Just thinking out loud.
Senior Member
#47
bhofmann wrote:Most open source projects use open source tools that are available to everyone (like MySQL). I think this is why most open source projects are successful in a community environment.

Depending on the complexity of the project you might be able to export table schemas and sample data that can be executed in a variety of databases. Then developers can have a development environment with the database of their choice. Proper application architecture requires that data access be a separate layer anyway. Of course there are slight differences in SQL syntax between some databases...

Just thinking out loud.
which is where middleware software like Hybernate come in which abstracts the database from the application. And now we have come full circle. ;)

Tom
Member
#48
tomw wrote:which is where middleware software like Hybernate come in which abstracts the database from the application. And now we have come full circle. ;)

Tom
Exactly! I think if there are enough participants a community project for the Church will still work. Even with the database issue.