Python Yield
Overview
Explain the common usage scenarios of iterators, generators, and yield fields in Python.
Iterators
Python object implements iter() and next() methods, we become iterable objects (iterables), through iter() can return an iterator object (iterators).
- __iter__() method: return the iterator object itself
- __next__() method: returns the next element of the container, and raises a StopIteration Exception at the end to terminate the iterator
1lst = [1, 2, 3]
2print(type(lst))
3new_iter = lst.__iter__()
4print(type(new_iter))
5
6# Output
7<class 'list'>
8<class 'list_iterator'>
The for loop actually gets iterators by iter() and then does next() to fetch until StopIteration.
Generators&yield
Generators are a special kind of iterators. If a field exists anywhere in the function, when you call the function, the function will not execute directly, but will return a generator. In addition generators also support generator expressions (similar to lists, except that [] is replaced with ()).
1def test():
2 print("for test")
3 yield 0
4gen1 = test()
5print(type(gen1))
6gen2 = (x*x for x in range(0, 3))
7print(type(gen2))
8
9# Output
10<class 'generator'>
11<class 'generator'>
- yield
Unlike iterators, which store all of their content in memory, generators allocate memory as next() is called over and over again.
Each time generators will run to the yield field, then save the generator state and return; the next call to next() will continue from the current position to the next yield field. This continues until StopIteration stops. Take a look at the following example:
1def test():
2 print("start")
3 yield 0
4 print("end")
5 yield 1
6
7gen1 = test()
8print(gen1.__next__())
9print(gen1.__next__())
10
11print("")
12gen2 = (x*x for x in range(0, 3))
13print(gen2.__next__())
14print(gen2.__next__())
15print(gen2.__next__())
16print(gen2.__next__())
17
18# Output
19start
200
21end
221
23
240
251
264
27Traceback (most recent call last):
28 File "/home/vagrant/aa.py", line 16, in <module>
29 print(gen2.__next__())
30StopIteration
Yield In Action
We have a batch of 100 pieces of data in a list and want to divide it into 10 groups of 10 and process them separately:
1def chunks(lst, n):
2 """Yield successive n-sized chunks from lst."""
3 for i in range(0, len(lst), n):
4 yield lst[i:i + n]
5
6origin_lst = list(range(0, 100))
7for i in chunks(origin_lst, 10):
8 print(i)
9
10# Output
11[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
12[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
13[20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
14[30, 31, 32, 33, 34, 35, 36, 37, 38, 39]
15[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
16[50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
17[60, 61, 62, 63, 64, 65, 66, 67, 68, 69]
18[70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
19[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]
20[90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
Reference Links: