Multithreading SQL connections

Hi there
I am working on the design of a system that its’ main function will be to
talk to devices over TCP/IP. There can be up to 50,000 devices that I need t
o
talk to within an hour period.
There will be a Queue on SQL 2005 which will contain requests for these
devices. I will need to check this queue and depending on the information on
the queue I need to talk to the devices. There will be one request for each
device every hour, therefore I need to be able to process multiple
communications at one time.
Here is my question: Should I create independent threads which would read
SQL and do pretty much everything, having a collection of about 50 or so
threads doing this? I have heard that I should avoid this high number of
connections to SQL at one time.
The other solution that I came up with is to have one SQL reader and have
the communication sockets as the threads, so they do their job and return. I
n
this thread communication scenario should I destroy the thread once I am
done, or should I have thread communication and keep the thread alive and
just send new requests to it?
I know this is pretty broad, but any help would be appreciated.
Thank you,
Spike S50 connections at a time in SQL Server is not an issue at all. Especially
if you are using connection pooling. When you say Queue what exactly do you
mean? Is this a regular table or is this a Queue from Service Broker?
Andrew J. Kelly SQL MVP
"Spike Spiegel" <cbb_spike@.hotmail.com> wrote in message
news:8AFEA5E5-1E9B-4D23-BFD9-1AEB25230AF6@.microsoft.com...
> Hi there
> I am working on the design of a system that its' main function will be to
> talk to devices over TCP/IP. There can be up to 50,000 devices that I need
> to
> talk to within an hour period.
> There will be a Queue on SQL 2005 which will contain requests for these
> devices. I will need to check this queue and depending on the information
> on
> the queue I need to talk to the devices. There will be one request for
> each
> device every hour, therefore I need to be able to process multiple
> communications at one time.
> Here is my question: Should I create independent threads which would read
> SQL and do pretty much everything, having a collection of about 50 or so
> threads doing this? I have heard that I should avoid this high number of
> connections to SQL at one time.
> The other solution that I came up with is to have one SQL reader and have
> the communication sockets as the threads, so they do their job and return.
> In
> this thread communication scenario should I destroy the thread once I am
> done, or should I have thread communication and keep the thread alive and
> just send new requests to it?
> I know this is pretty broad, but any help would be appreciated.
> Thank you,
> Spike S|||Hi Andrew
Sorry, I should have not used the word queue. It is basically just a table
with a very log record, about 50 fields, my application would be pulling fro
m
it.
What do you mean by "connection pooling?"
Right I am running a test and basically I am opening a connection using the
System.Data.SqlClient.SqlConnection class, and reading my record with
SqlConnection + SqlDataReader and writing records with the SqlConnection.+
TSQL statement. Are these the most efficient/fastest way to do this? I am
very interested in speed here.
If I were to create a thread and send the SqlConnection to the thread for
them all to share the same connection, would that make the code more
efficient?
What would be a good threshold for number of threads? 100? 500? 1000? 10000?
Please let me know.
Thank you,
Spike S.|||> What do you mean by "connection pooling?"
I would google for more details but essentially it is used by .net to make
the process of connecting and disconnecting much more efficient. You retain
a pool of connections that stay connected all the time. These are then
handed out as the app requests new connections and instead of closing it
completely it just cleans it up and makes it ready for the next user. It's
the default behavior with .net.

> Right I am running a test and basically I am opening a connection using
> the
> System.Data.SqlClient.SqlConnection class, and reading my record with
> SqlConnection + SqlDataReader and writing records with the SqlConnection.+
> TSQL statement. Are these the most efficient/fastest way to do this? I am
> very interested in speed here.
Actually using stored procedures to read and write the data is the most
efficient way.

> What would be a good threshold for number of threads? 100? 500? 1000?
> 10000?
Testing is the only way to know for sure.
Andrew J. Kelly SQL MVP
"Spike Spiegel" <cbb_spike@.hotmail.com> wrote in message
news:96A7733E-31CD-4744-9DD9-FD8CF64278A0@.microsoft.com...
> Hi Andrew
> Sorry, I should have not used the word queue. It is basically just a table
> with a very log record, about 50 fields, my application would be pulling
> from
> it.
> What do you mean by "connection pooling?"
> Right I am running a test and basically I am opening a connection using
> the
> System.Data.SqlClient.SqlConnection class, and reading my record with
> SqlConnection + SqlDataReader and writing records with the SqlConnection.+
> TSQL statement. Are these the most efficient/fastest way to do this? I am
> very interested in speed here.
> If I were to create a thread and send the SqlConnection to the thread for
> them all to share the same connection, would that make the code more
> efficient?
> What would be a good threshold for number of threads? 100? 500? 1000?
> 10000?
> Please let me know.
> Thank you,
> Spike S.|||Hi Andrew
Thank you very much for all the information.
I did some testing, using the connection pooling, and it is exactly what I
was looking for, I can have around 50 threads before I see a performance hit
.
This is the first time I posted into a managed Microsoft newsgroup. From
this experience I will not think twice before checking the managed group
again.
Thank you again for all your help.
Take care,
Spike S.|||Please note that Andy doesn't have anything to do with the MSDN managed
newsgroup program/policy/whatever you want to call it. Andy doesn't work for
Microsoft and is out here helping everybody on his own dime. That, in my
opinion, means he deserves even more thanks. :-)
Sincerely,
Steve Dybing
This posting is provided "AS IS" with no warranties, and confers no rights.
"Spike Spiegel" <cbb_spike@.hotmail.com> wrote in message
news:74D54EA9-786B-45AE-8007-88804D2A4B56@.microsoft.com...
> Hi Andrew
> Thank you very much for all the information.
> I did some testing, using the connection pooling, and it is exactly what I
> was looking for, I can have around 50 threads before I see a performance
> hit.
> This is the first time I posted into a managed Microsoft newsgroup. From
> this experience I will not think twice before checking the managed group
> again.
> Thank you again for all your help.
> Take care,
> Spike S.|||Double thanks to Andrew then...|||Double you are welcome.
Andrew J. Kelly SQL MVP
"Spike Spiegel" <cbb_spike@.hotmail.com> wrote in message
news:A5A1E014-F5C1-43E4-8956-F9D19EA5C153@.microsoft.com...
> Double thanks to Andrew then...|||Ahhh gee Thanks Steve :)
Andrew J. Kelly SQL MVP
"Steve Dybing [MSFT]" <steve.dybing@.online.microsoft.com> wrote in message
news:eFQhpyFdGHA.3908@.TK2MSFTNGP04.phx.gbl...
> Please note that Andy doesn't have anything to do with the MSDN managed
> newsgroup program/policy/whatever you want to call it. Andy doesn't work
> for Microsoft and is out here helping everybody on his own dime. That, in
> my opinion, means he deserves even more thanks. :-)
> --
> Sincerely,
> Steve Dybing
> This posting is provided "AS IS" with no warranties, and confers no
> rights.
> "Spike Spiegel" <cbb_spike@.hotmail.com> wrote in message
> news:74D54EA9-786B-45AE-8007-88804D2A4B56@.microsoft.com...
>sql