Architecture options for running a long process from an ASP.NET website

Hi,

We currently have a solution where a user can upload a file, which gets processed in an ASP.NET v1.1 intranet website

The solution was designed to handle 10-100 items, but customers have (ab)used the solution to process 1000-10000 items. Processing 100 items takes less then a minute, processing 1000 items takes around 7 minutes, larger uploads take (non-linear) longer.

Now besides performance tuning the processing, I want to gather feedback on what is the best strategy to take the upload and have it process 'asynchronously'. After the upload, the user would immediatly get a status page which would refresh itself (every so often).

So I have a few questions about what would be the best route to go about

1) Which process to host the 'async processing' in
options I consider: a custom .NET Windows Service or a Web Service
2) How to pass the uploaded file to the 'async processing'
options:
- shared drive location: copy the file to a shared location, have the process pick it up
- Web Service 'File Upload':
- ftp: firewall support
- MSMQ: guaranteed delivery
3) How to communicate between the intranet web-site and the 'async processing'
options: .NET Remoting - Web Service calls
ofcourse my assumption here is that .NET Remoting goes with the .NET Windows Service and the Web Service calls with the web service.
4) Security considerations, Exception Handling and Logging
=> Enterprise Library

Any comments, feedback is welcome. Blogs, articles, books, even code

Thanks,

Rudi

Architecture5

Hi,
You have almost suggested the solution, what I would like to use assuming that i have a database as well is...

Solution Vision:
Create a web service to upload the file. Put it in some shared folder at server and log its path in the database, give it some status for example pending. Make a window service that get the file location from the database and start processing the file change status of file to processing during processing and after process set status to processed. Create a web service to get status of file from the database so that Intranet web site can any time and see status.

Process:

Step1: File Upload
Create a web service for the file upload. Make a folder on the server in which you will upload it. After uploading the file rename it to unique id (GUI ID), now store both original name (if needed) and new name (i.e. GUI ID) in a database table and give it status Pending.
Following could be the table fields, You can change it according to your preference.


FileNameNew	New filename
FileNameOrignal	Old file name
Status	Current status
StatusUpdated	Status last update time

Step2: File Processing
Create a window service that will be checking database for an new files. If new file record found then get file name from the database and you can get shared folder path from the app.config of the window service after adding key and setting it value (e.g. D:\SYS\UPLOADS). Now create a queue in window service and implement threading. You should also set some max parallel threads count so that if more than 10 or 15 files found system not get stuck, you should process a fixed number of files at a time in parallel threads for the system performance concerns.

Step3: Status Report on intranet website
Create web service to get status of the file from database and show. More if status is processed the you can show files items as well or what ever your logic is.

Step4: Security/Exception Handling
You can use log4net (free logging component) at in web service and window service for exception logging.

Check file for viruses after uploading.

Use web.config of web service to allow authorize users.

Hope this will help.

Thanks for the reply. Good stuff.

I have also found an interesting article: http://msdn.microsoft.com/msdnmag/issues/05/03/SchedulingASPNETCode/

Since the processing is very SQL Server intensive, I'm also wondering if there are other options to communicate the progress, other than the database.

Hi,

Actuly processing is not database oriented but storage of data for central management.
SQL Server is just for example, you can use any database.
There are many free as well e.g. mySQL

Catalog of free database systems!

by the way link you have mentioned also use database.

More database is an easy way to centeralize the data.

Best of Luck!

Have you considered a BizTalk orchestration exposed as webservice as one of the options It is something you can consider. BizTalk orchestrations provide all of the below capabilites.

1. Handling file based inputs

2. Communication with/as web services

3. Synchronous/Asynchronous communications

4. Reliable messaging services

If you are doing asynchronous work in a database, you need to consider using Service Broker in SQL Server 2005 to manage the work. You can queue the work to be processed on a Service Broker queue. Service Broker will manage the processing and provide a mechanism for returning status asynchronously to the caller. Because it is built into the database, the database transactionally provides reliability and will ensure that the tasks are processed in spite of any failures. Combined with Database Mirroring, Service Broker can provide unprecidented levels of reliability in asynchronous processing.

Service broker & Mirroring are indeed a powerful combination - I do suggest you benchmark them together first to make sure the performance meets your need (on your target platform)

Arnon

Hi,

Thanks all for the extra info, but I should clarify a few things:

The processing reads the information from the file and inserts the records into the database using a simple (extensible) pipeline.

=> this is where the heavy part of the logic is. 70% of the processing time is related to the statements executed against the SQL Server.

1) Mubshir, that is why I was looking for an alternative way to communicate the status updates. I do not want to add extra load on the SQL Server.

2) Sarit, we did evaluate using BizTalk as the central piece for our solution. but it was decided that we were not to use BizTalk (extra licence cost).

3) Roger, I wish I could go full out on using SQL 2005, but that won't be for the near future. We are tied to supporting SQL 2000. (most likely until all of our customers have moved on)

The solution really needs to be 'old' skool, .NET v1.1 with SQL 2000. But I want to enable the architecture to port to new technology as smoothly as possible, if possible.

Meanwhile, I have tuned the process more and found/fixed a great big bottleneck, but inherently the solution is not fit to scale, and sales have found this tool to be the next ETL solution. I strongly object, but sales will be sales.

Thanks again for the feedback.