<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Initial conditions
Step1:  Generate a sequence
Generate long sequence of real numbers
For further analysis it's handy to convert it to binary array
Then we can get different sequences of integers by splitting binary array into blocks
Step2:  Entropies
We need to calculate different entropies for each sequence
Step3:  Now we have mappings from binary block size to entropies of sequences
Step4:  Now we will calculate average entropy for each block size
Step5:  Get same entropies for chaotic sequence
Sequences of uniformly distributed numbers with same binary sizes as existent sequences have the biggest possible entropies
Step6:  To get only needed numbers we need to calculate standard deviations
It will show us how good the average value estimates entropy of the source
Step7:  Analysis
We can see from the chart that standard deviation grows to some moment and then falls down
Experiments with other source sequences have shown that the point with maximal standard deviation moves to right with sequence size growth
This can mean that before that moment we can gather data which is good for entropy estimate and after that we have not enough numbers for estimate
For example, when block size is equal to sequence length, entropy will be zero but source generates different pseudorandom numbers
Estimate
Entropy of the source should be calculated as
H = lim(n to infinity) Hn/n
where n is a block size and Hn is an entropy of the sequence with size of block equal to n
Here are following conserns
Step8:  Results
Here are estimate of the source entropy and value of chaotic source
<ASSISTANT_TASK:>
Python Code:
MU = 3.9
N = int(10E4)
INITIAL = 0.5
MIN_SIZE = 2
MAX_SIZE = 26
BITS_RANGE = array(list(range(MIN_SIZE, MAX_SIZE + 1)))
def generate(x, mu, n):
    current = x
    for _ in range(n):
        yield current
        current = mu * current * (1 - current)
def bin_to_dec(sequence, bits):
    aligned_sequence = sequence.flatten()[:sequence.size - sequence.size % bits]
    binary_matrix = aligned_sequence.reshape((sequence.size // bits, bits))
    result_sequence = array([binary_matrix[:,i] * (2**i) for i in range(binary_matrix.shape[1])])
    return result_sequence.sum(axis=0)
def R(probabilities, beta):
    return -log((probabilities[probabilities>0]**beta).sum()) / (beta - 1)
def H(probabilities):
    indices = probabilities>0
    return -(log(probabilities[indices]) * probabilities[indices]).sum()
def show_entropies(entropies, errors=None, count=None, labels=None):
    for entropy, label in zip(entropies, labels):
        if count is None:
            plt.plot(BITS_RANGE, entropy, '-o', label=label)
        else:
            plt.plot(BITS_RANGE[:count], entropy, '-o', label=label)
    if errors is not None:
        plt.errorbar(**errors)
    plt.legend()
    plt.show()
Explanation: Initial conditions
End of explanation
source_sequence = array(list(generate(INITIAL, MU, N)))
bin_sequence = (source_sequence > 0.5).astype('i')
sequences = array([bin_to_dec(bin_sequence, bits_count).astype('i') for bits_count in BITS_RANGE])
probabilities = array([bincount(sequence)/sequence.size for sequence in sequences])
probabilities[1]
plt.plot(source_sequence[:100])
plt.show()
Explanation: Generate a sequence
Generate long sequence of real numbers
For further analysis it's handy to convert it to binary array
Then we can get different sequences of integers by splitting binary array into blocks
End of explanation
shannon_entropies = [H(p) for p in probabilities]
renyi_entropies2 = [R(p, 2) for p in probabilities]
renyi_entropies3 = [R(p, 3) for p in probabilities]
Explanation: Entropies
We need to calculate different entropies for each sequence
End of explanation
show_entropies([shannon_entropies, renyi_entropies2, renyi_entropies3], labels=['Shannon', 'Renyi (2)', 'Renyi (3)'])
Explanation: Now we have mappings from binary block size to entropies of sequences
End of explanation
means = array([shannon_entropies, renyi_entropies2, renyi_entropies3]).mean(axis=0)
show_entropies([shannon_entropies, renyi_entropies2, renyi_entropies3, means], labels=['Shannon', 'Renyi (2)', 'Renyi (3)', 'Mean'])
Explanation: Now we will calculate average entropy for each block size
End of explanation
shannon_entropies_chaos = [n*log(2) for n in BITS_RANGE]
renyi_entropies2_chaos = [-(n*log(2)-2*n*log(2)) for n in BITS_RANGE]
renyi_entropies3_chaos = [-(n*log(2)-3*n*log(2))/2 for n in BITS_RANGE]
show_entropies([shannon_entropies, renyi_entropies2, renyi_entropies3, means, shannon_entropies_chaos], labels=['Shannon', 'Renyi (2)', 'Renyi (3)', 'Mean', 'Uniform entropy'])
Explanation: Get same entropies for chaotic sequence
Sequences of uniformly distributed numbers with same binary sizes as existent sequences have the biggest possible entropies
End of explanation
standard_deviation = array([shannon_entropies, renyi_entropies2, renyi_entropies3]).std(axis=0)
show_entropies([shannon_entropies, renyi_entropies2, renyi_entropies3], labels=['Shannon', 'Renyi (2)', 'Renyi (3)'], errors={'x': BITS_RANGE, 'y': means, 'yerr': standard_deviation, 'label': 'Mean'})
Explanation: To get only needed numbers we need to calculate standard deviations
It will show us how good the average value estimates entropy of the source
End of explanation
count = standard_deviation.argmax()
a, b = lstsq(vstack([BITS_RANGE[:count], ones(count)]).T, means[:count])[0]
approximation_error = array([means[:count], a*BITS_RANGE[:count]+b]).std(axis=0)
show_entropies([shannon_entropies[:count], renyi_entropies2[:count], renyi_entropies3[:count], means[:count]], labels=['Shannon', 'Renyi (2)', 'Renyi (3)', 'Mean'], errors={'x': BITS_RANGE[:count], 'y': a*BITS_RANGE[:count]+b, 'yerr': approximation_error, 'label': 'Linear approximation'}, count=count)
show_entropies([shannon_entropies_chaos[:count], means[:count]], labels=['Uniform entropy', 'Mean'], errors={'x': BITS_RANGE[:count], 'y': a*BITS_RANGE[:count]+b, 'yerr': approximation_error, 'label': 'Linear approximation'}, count=count)
Explanation: Analysis
We can see from the chart that standard deviation grows to some moment and then falls down
Experiments with other source sequences have shown that the point with maximal standard deviation moves to right with sequence size growth
This can mean that before that moment we can gather data which is good for entropy estimate and after that we have not enough numbers for estimate
For example, when block size is equal to sequence length, entropy will be zero but source generates different pseudorandom numbers
Estimate
Entropy of the source should be calculated as
H = lim(n to infinity) Hn/n
where n is a block size and Hn is an entropy of the sequence with size of block equal to n
Here are following conserns:
- we cannot get n big enough without sequence size increase;
- it's impossible to get very big sequence;
- estimate based on single block size will have undefined error
We can use least squares method to estimate the mean entropies (will be denoted as Hn') using linear model
Hn' = k * n + c
Here k will be an estimate of H because following equality will take a place when n is big enough
Hn'/n = k + c/n -&gt; k = H
End of explanation
print('Source entropy estimate is', a)
print('Chaotic entropy is', log(2))
(array(shannon_entropies[1:]) - shannon_entropies[:-1])[:count][:6]
Explanation: Results
Here are estimate of the source entropy and value of chaotic source
End of explanation
<END_TASK>