Programming
multithreading python-3.x queue late-binding
Updated Sun, 24 Jul 2022 21:21:54 GMT

Is python's queue.Queue.put() method asynchronous?


If I run a thread with the following function as the worker,

q = queue.Queue()
def worker():
    while True:
        t = {}
        for i in range(3):
            t['a'] = i
            q.put(t)

the queue is populated with dictionaries that are all the same, i.e., {'a': 2} instead of the sequence {'a': 0}, {'a': 1}, {'a': 2}. I assume this is because the put() method runs after the for loop has finished and the last value of i was 2. Am I interpreting that right?

Now, if I move the instantiation of the dictionary inside the for loop,

def worker():
    while True:
        for i in range(3):
            t = {'a': i}
            q.put(t)

the queue is populated with the desired sequence. My interpretation is that in the first instance, I create a dictionary object in memory, then begin a for loop and reassign its value 3 times but the put() calls happen after the loop has finished. In the second instance, I create a new dictionary object every iteration of the for loop and so when the put() calls occur after the loop, they access 3 distinct instances of the dictionary with their own key-value pairs.

Can anyone shed some light on what's happening behind the curtain here?




Solution

Am I interpreting that right?

You observe such behavior because youre modifying the same object all the time

Lets put aside queues / threads and run a simplified equivalent of your code with some prints to understand whats happening

t = {}
l = []
for i in range(3):
    t['a'] = i
    l.append(t)
print(l)
t['a'] = 20
print(l)
print(map(id, l))
[{'a': 2}, {'a': 2}, {'a': 2}]
[{'a': 20}, {'a': 20}, {'a': 20}]
# they are all the same!
[4474861840, 4474861840, 4474861840]

So it has nothing to do we threads/queues - youre just adding the same object 3 times.

Now, if I move the instantiation of the dictionary inside the for loop

In this case you create a new object every time like in the following code:

l = []
for i in range(3):
    t = {}
    t['a'] = i
    l.append(t)
print(l)
t['a'] = 20
print(l)
print(map(id, l))
[{'a': 0}, {'a': 1}, {'a': 2}]
[{'a': 0}, {'a': 1}, {'a': 20}]
# they are all different!
[4533475600, 4533502592, 4533502872]

So no magic here

back to your question

This is what could be of interest for you: Is pythons queue.Queue.put() thread safe? meaning that the global variable q could be accessed by multiple concurrent threads safely. The answer is yes - it is thread safe

The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics







External Links

External links referenced by this document: