Paginate Safes pending to be processed #2616

moisses89 · 2025-08-18T10:37:50Z

Description

The query pending Safes to be processed can be slow in case of a big amount of Safes pending.
The idea of this PR is paginate the query without a distinct to avoid a big sort on disk.
The distinct is being appliead in redis, with the following features:

use O(1) operations, to ensure have not big blockers.
expiry on task timeou, to avoid acumulate keys on redis
remove with unlink if the task is succesfully ending, at this way redis will do the delete on bakcgroung avoiding a big block.

safe_transaction_service/history/models.py

After analyzing usage of indexes in production: - `call_type`, `tx_type` and `refund_address` indexes are never used. - `value`, `to`, `timestamp` and `block_number` are close to no usage. - `_from` index is covered in `("_from", "timestamp", "id")` index Raw data about index usage in Mainnet and Worldchain production databases: ``` mainnet => select schemaname || '.' || indexrelname as index, idx_scan, last_idx_scan from pg_stat_user_indexes; index | idx_scan | last_idx_scan ------------------------------------------------------------------------+------------+------------------------------- public.account_abstraction_safeoperation_pkey | 3305 | 2025-09-03 11:32:24.308323+00 public.account_abstraction_safeoperation_user_operation_id_key | 194047 | 2025-09-03 12:16:44.046243+00 public.account_abstraction_safeoperationconfirmation_pkey | 0 | public.account_abstraction_useroperation_pkey | 393064 | 2025-09-03 12:16:44.046243+00 public.account_abstraction_useroperationreceipt_pkey | 0 | public.account_abstraction_useroperationreceipt_user_operation_id_key | 196116 | 2025-09-03 12:16:44.046243+00 public.auth_group_name_key | 0 | public.auth_group_permissions_group_id_permission_id_0cd325b0_uniq | 0 | public.auth_group_permissions_pkey | 0 | public.auth_group_pkey | 0 | public.auth_permission_content_type_id_codename_01ab375a_uniq | 824 | 2025-09-03 06:07:39.418652+00 public.auth_permission_pkey | 0 | public.auth_user_groups_pkey | 0 | public.auth_user_groups_user_id_group_id_94350c0c_uniq | 0 | public.auth_user_pkey | 494 | 2025-08-17 10:16:40.55475+00 public.auth_user_user_permissions_pkey | 0 | public.auth_user_user_permissions_user_id_permission_id_14a6b632_uniq | 0 | public.auth_user_username_key | 0 | public.authtoken_token_pkey | 0 | public.authtoken_token_user_id_key | 0 | public.contracts_contract_pkey | 120471052 | 2025-09-03 12:16:59.115745+00 public.contracts_contractabi_abi_hash_key | 265 | 2025-09-03 07:00:03.682091+00 public.contracts_contractabi_pkey | 359092536 | 2025-09-03 12:16:59.115745+00 public.django_admin_log_pkey | 0 | public.django_celery_beat_clockedschedule_pkey | 119 | 2025-08-05 09:01:39.512435+00 public.django_celery_beat_crontabschedule_pkey | 4020 | 2025-07-20 11:03:35.871801+00 public.django_celery_beat_intervalschedule_pkey | 4796 | 2025-07-20 11:03:35.871801+00 public.django_celery_beat_periodictask_name_key | 0 | public.django_celery_beat_periodictask_pkey | 218686 | 2025-09-03 12:14:14.418198+00 public.django_celery_beat_periodictasks_pkey | 0 | public.django_celery_beat_solar_event_latitude_longitude_ba64999a_uniq | 0 | public.django_celery_beat_solarschedule_pkey | 0 | public.django_content_type_app_label_model_76bd3d3b_uniq | 280 | 2025-07-18 09:08:13.181833+00 public.django_content_type_pkey | 69 | 2025-08-05 08:28:22.074872+00 public.django_migrations_pkey | 0 | public.django_session_pkey | 0 | public.django_site_domain_a2e37b91_uniq | 0 | public.django_site_pkey | 2 | 2025-07-18 09:47:57.483885+00 public.history_chain_pkey | 0 | public.history_erc20transfer_pkey | 2086000 | 2025-09-03 12:16:41.251894+00 public.history_erc721transfer_pkey | 115385 | 2025-09-03 10:43:20.857081+00 public.history_ethereumblock_block_hash_key | 1042640 | 2025-09-03 12:16:55.593524+00 public.history_ethereumblock_parent_hash_key | 188236 | 2025-09-03 12:16:55.593524+00 public.history_ethereumblock_pkey | 3928224181 | 2025-09-03 12:16:59.115745+00 public.history_ethereumtx_pkey | 3917975592 | 2025-09-03 12:16:59.115745+00 public.history_indexingstatus_pkey | 55018 | 2025-09-03 12:16:45.881564+00 public.history_internaltx_pkey | 612701168 | 2025-09-03 12:16:55.672998+00 public.history_internaltxdecoded_pkey | 184120464 | 2025-09-03 12:16:55.672998+00 public.history_moduletransaction_pkey | 1110867 | 2025-09-03 12:16:53.099946+00 public.history_multisigconfirmation_pkey | 87760 | 2025-09-03 12:08:05.345648+00 public.history_multisigtransaction_pkey | 102115239 | 2025-09-03 12:16:59.270002+00 public.history_proxyfactory_pkey | 48 | 2025-07-21 10:26:11.91124+00 public.history_safecontract_pkey | 60773876 | 2025-09-03 12:16:59.262085+00 public.history_safecontractdelegate_pkey | 348 | 2025-09-03 10:27:05.938372+00 public.history_safelaststatus_internal_tx_id_key | 27568 | 2025-09-03 09:41:00.651579+00 public.history_safelaststatus_pkey | 102883303 | 2025-09-03 12:16:56.991618+00 public.history_safemastercopy_pkey | 0 | public.history_saferelevanttran_ethereum_tx_id_safe_096ba90b_uniq | 27399869 | 2025-09-03 12:16:55.593524+00 public.history_saferelevanttransaction_pkey | 4397764 | 2025-09-03 12:16:41.251894+00 public.history_safestatus_pkey | 54392 | 2025-09-03 09:41:00.651579+00 public.safe_messages_safemessage_pkey | 348468 | 2025-09-03 12:16:53.370564+00 public.safe_messages_safemessageconfirmation_pkey | 0 | public.tokens_token_pkey | 47804773 | 2025-09-03 12:16:59.065353+00 public.tokens_tokenlist_pkey | 0 | public.tokens_tokenlist_url_key | 0 | public.tokens_tokennotvalid_pkey | 3723 | 2025-09-03 12:09:35.739603+00 public.unique_erc20_transfer_index | 11603870 | 2025-09-03 12:16:58.371675+00 public.unique_erc721_transfer_index | 5915733 | 2025-09-03 12:16:58.569723+00 public.unique_internal_tx_trace_address | 20462277 | 2025-09-03 12:16:55.593524+00 public.unique_multisig_transaction_owner_confirmation | 34166906 | 2025-09-03 12:16:55.365359+00 public.unique_safe_contract_delegate_delegator | 540 | 2025-09-03 11:29:24.010734+00 public.unique_safe_message_confirmation_owner | 11527329 | 2025-09-03 12:12:15.440001+00 public.unique_safe_operation_owner_confirmation | 0 | public.unique_safe_tx_address_status | 18857847 | 2025-09-03 12:15:55.73164+00 public.account_abs_sender_663a75_idx | 0 | public.account_abstraction_safeop_safe_operation_id_0bd247f6 | 1779 | 2025-09-03 11:37:19.920681+00 public.account_abstraction_safeop_signature_type_43748fb5 | 0 | public.account_abstraction_safeoperation_module_address_3646c35b | 0 | public.account_abstraction_useroperation_entry_point_2fc2134f | 0 | public.account_abstraction_useroperation_ethereum_tx_id_fd4ae6ec | 101938 | 2025-09-03 12:16:44.046243+00 public.account_abstraction_useroperation_paymaster_751e7bc1 | 0 | public.account_abstraction_useroperation_sender_d9a3df38 | 4 | 2025-08-15 18:44:08.992015+00 public.auth_group_name_a6ea08ec_like | 0 | public.auth_group_permissions_group_id_b120cbf9 | 0 | public.auth_group_permissions_permission_id_84c5c92e | 0 | public.auth_permission_content_type_id_2f476e4b | 16 | 2025-07-18 09:08:13.181833+00 public.auth_user_groups_group_id_97559544 | 0 | public.auth_user_groups_user_id_6a12ed8b | 0 | public.auth_user_user_permissions_permission_id_1fbb5f2c | 0 | public.auth_user_user_permissions_user_id_a95ead1b | 0 | public.auth_user_username_6821ab7c_like | 15 | 2025-08-12 11:59:19.941222+00 public.authtoken_token_key_10f0b77e_like | 14 | 2025-09-02 06:05:57.755956+00 public.contracts_contract_contract_abi_id_1c37b4bd | 240308852 | 2025-09-03 12:16:59.115745+00 public.django_admin_log_content_type_id_c4bce8eb | 0 | public.django_admin_log_user_id_c564eba6 | 12 | 2025-07-31 12:56:07.277776+00 public.django_celery_beat_periodictask_clocked_id_47a69f82 | 8535 | 2025-09-03 12:13:10.964668+00 public.django_celery_beat_periodictask_crontab_id_d3cba168 | 0 | public.django_celery_beat_periodictask_interval_id_a8ca27da | 0 | public.django_celery_beat_periodictask_name_265a36b7_like | 110066 | 2025-09-03 12:14:14.418198+00 public.django_celery_beat_periodictask_solar_id_a87ce72c | 0 | public.django_session_expire_date_a5c62663 | 0 | public.django_session_session_key_c0390e0f_like | 433 | 2025-08-17 10:16:40.55475+00 public.django_site_domain_a2e37b91_like | 0 | public.history_erc20transfer_ethereum_tx_id_16a6036b | 521594 | 2025-09-03 12:16:58.569723+00 public.history_erc20transfer_timestamp_b14c5ef4 | 17954 | 2025-09-03 12:00:08.795607+00 public.history_erc721transfer_ethereum_tx_id_aa020110 | 499 | 2025-09-03 11:05:20.138249+00 public.history_erc721transfer_timestamp_aaadcf46 | 1725 | 2025-09-03 10:03:19.548065+00 public.history_erc__from_64986c_idx | 141 | 2025-09-03 10:03:19.548065+00 public.history_erc__from_72fb41_idx | 141 | 2025-09-03 10:03:19.548065+00 public.history_erc__from_d88198_idx | 11942685 | 2025-09-03 12:16:59.065353+00 public.history_erc__from_d919fc_idx | 5903606 | 2025-09-03 12:16:54.879351+00 public.history_erc_address_94cee3_idx | 126 | 2025-09-03 10:03:19.548065+00 public.history_erc_address_dba64d_idx | 924 | 2025-09-03 12:09:49.631519+00 public.history_erc_to_02d4ab_idx | 1578 | 2025-09-03 10:03:19.548065+00 public.history_erc_to_246e8e_idx | 5926707 | 2025-09-03 12:16:54.879351+00 public.history_erc_to_d39948_idx | 12193436 | 2025-09-03 12:16:59.065353+00 public.history_erc_to_f32154_idx | 10094 | 2025-09-03 12:00:08.795607+00 public.history_ethereumtx__from_e36d4662 | 0 | public.history_ethereumtx_block_id_92e7f70e | 119925645 | 2025-09-03 12:16:59.115745+00 public.history_ethereumtx_status_d1d0de2c | 0 | public.history_ethereumtx_to_9fb37c13 | 0 | public.history_indexingstatus_block_number_40805e1c | 29816 | 2025-09-03 12:16:41.251894+00 public.history_int_to_e72886_idx | 28 | 2025-09-01 11:41:26.842004+00 public.history_internal_transfer_from | 6766333 | 2025-09-03 12:16:59.065353+00 public.history_internal_transfer_idx | 6960650 | 2025-09-03 12:16:59.065353+00 public.history_internaltx__from_36470cfd | 1013954 | 2025-09-02 22:02:12.827141+00 public.history_internaltx_block_number_9004df5f | 63 | 2025-08-02 19:04:18.743767+00 public.history_internaltx_call_type_1f0c4296 | 0 | public.history_internaltx_contract_address_134466b3 | 99042 | 2025-09-03 12:16:44.046243+00 public.history_internaltx_ethereum_tx_id_e6ac35ab | 5615799 | 2025-09-03 12:16:58.569723+00 public.history_internaltx_refund_address_09530831 | 0 | public.history_internaltx_timestamp_ef4d64cb | 18198 | 2025-09-03 12:00:08.795607+00 public.history_internaltx_tx_type_bb97604e | 0 | public.history_internaltx_value_idx | 41 | 2025-09-02 13:45:00.266275+00 public.history_internaltxdecoded_function_name_333ae03e | 61 | 2025-08-01 15:18:15.677309+00 public.history_internaltxdecoded_function_name_333ae03e_like | 0 | public.history_moduletransaction_module_5bf50dd1 | 3478 | 2025-09-03 12:00:11.087972+00 public.history_moduletransaction_safe | 2222160 | 2025-09-03 12:16:56.337482+00 public.history_moduletransaction_to_041e4c7b | 0 | public.history_multisigconfirmation_ethereum_tx_id_71e0ab73 | 4948 | 2025-09-03 09:41:00.651579+00 public.history_multisigconfirmation_multisig_transaction_hash_4c8cbf7a | 16785034 | 2025-09-03 12:16:52.35018+00 public.history_multisigconfirmation_multisig_transaction_id_43ec94d5 | 1667539923 | 2025-09-03 12:16:59.115745+00 public.history_multisigconfirmation_signature_type_9679d821 | 0 | public.history_multisigtransaction_ethereum_tx_id_c8d3f1cf | 127347749 | 2025-09-03 12:16:59.115745+00 public.history_multisigtransaction_failed_4842ad6b | 80 | 2025-08-28 20:28:11.261355+00 public.history_multisigtransaction_nonce_a98eecaa | 16705965 | 2025-09-03 12:16:57.649154+00 public.history_multisigtransaction_safe_ba8bae68 | 141624951 | 2025-09-03 12:16:59.115745+00 public.history_multisigtransaction_to_e544fdbe | 79626 | 2025-09-03 12:14:51.933328+00 public.history_multisigtransaction_trusted_f8ea7938 | 29 | 2025-09-03 00:00:00.80496+00 public.history_multisigtx_safe_sorted | 114729846 | 2025-09-03 12:16:59.270002+00 public.history_proxyfactory_tx_block_number_51be6689 | 18 | 2025-07-21 02:21:01.221613+00 public.history_saf_address_aa71bd_idx | 6967101508 | 2025-09-03 12:16:59.115745+00 public.history_saf_enabled_020fea_gin | 177252 | 2025-09-03 12:16:58.02252+00 public.history_saf_owners_5fc97e_gin | 21621213 | 2025-09-03 12:16:57.597744+00 public.history_saf_safe_3768a5_idx | 19963367 | 2025-09-03 12:16:59.23732+00 public.history_safe_banned_idx | 109762 | 2025-08-08 08:36:31.465401+00 public.history_safecontract_created_a7d29917 | 306197 | 2025-09-03 12:16:55.183611+00 public.history_safecontract_ethereum_tx_id_4f63f583 | 2635 | 2025-09-03 09:41:00.651579+00 public.history_safecontractdelegate_delegate_440a3b28 | 80918 | 2025-09-03 12:13:13.46714+00 public.history_safecontractdelegate_delegator_9fd00422 | 1200925 | 2025-09-03 12:16:52.35018+00 public.history_safecontractdelegate_expiry_date_18014ead | 1201 | 2025-09-03 12:00:00.017333+00 public.history_safecontractdelegate_safe_contract_id_4389cdbf | 5195850 | 2025-09-03 12:16:58.241609+00 public.history_safemastercopy_tx_block_number_c1b3ce4f | 0 | public.history_saferelevanttransaction_ethereum_tx_id_0b704cf3 | 5704 | 2025-09-03 10:03:19.548065+00 public.history_safestatus_address_68cd154d | 5835 | 2025-09-02 16:01:08.607166+00 public.safe_messages_safemessage_safe_fc7e3fe6 | 4744108 | 2025-09-03 12:16:57.780849+00 public.safe_messages_safemessageconfirmation_owner_b930d28f | 17066 | 2025-09-03 12:08:05.345648+00 public.safe_messages_safemessageconfirmation_safe_message_id_1796dddc | 3914977 | 2025-09-03 12:16:57.780849+00 public.safe_messages_safemessageconfirmation_signature_type_34a29b90 | 0 | public.token_events_bugged_idx | 0 | public.token_spam_idx | 35929 | 2025-09-03 12:16:07.118271+00 public.token_trusted_idx | 639395 | 2025-09-03 12:16:58.241609+00 public.tokens_token_decimals_7c745c31 | 36 | 2025-09-03 12:09:51.046985+00 public.tokens_tokenlist_url_09b6a089_like | 0 | public.history_block_confirmed_idx | 41903 | 2025-09-03 12:16:00.010048+00 public.history_ethereumblock_timestamp_85f398f6 | 2402 | 2025-09-03 12:00:00.143555+00 public.history_int__from_2301f6_idx | 23582 | 2025-09-03 12:16:55.672998+00 public.history_decoded_processed_idx | 108557 | 2025-09-03 12:16:55.672998+00 public.history_decoded_not_proc_idx | 41637809 | 2025-09-03 12:16:55.672998+00 (179 rows) worldchain => select schemaname || '.' || indexrelname as index, idx_scan, last_idx_scan from pg_stat_user_indexes; index | idx_scan | last_idx_scan ------------------------------------------------------------------------+-----------+------------------------------- public.history_block_confirmed_idx | 8658 | 2025-09-02 12:03:00.011593+00 public.account_abstraction_safeoperation_pkey | 0 | public.account_abstraction_safeoperation_user_operation_id_key | 0 | public.account_abstraction_safeoperationconfirmation_pkey | 0 | public.account_abstraction_useroperation_pkey | 0 | public.account_abstraction_useroperationreceipt_pkey | 0 | public.account_abstraction_useroperationreceipt_user_operation_id_key | 0 | public.auth_group_name_key | 0 | public.auth_group_permissions_group_id_permission_id_0cd325b0_uniq | 0 | public.auth_group_permissions_pkey | 0 | public.auth_group_pkey | 0 | public.auth_permission_content_type_id_codename_01ab375a_uniq | 130 | 2025-09-01 11:18:45.836254+00 public.auth_permission_pkey | 0 | public.auth_user_groups_pkey | 0 | public.auth_user_groups_user_id_group_id_94350c0c_uniq | 0 | public.auth_user_pkey | 0 | public.auth_user_user_permissions_pkey | 0 | public.auth_user_user_permissions_user_id_permission_id_14a6b632_uniq | 0 | public.auth_user_username_key | 0 | public.authtoken_token_pkey | 0 | public.authtoken_token_user_id_key | 0 | public.contracts_contract_pkey | 1214 | 2025-09-02 12:00:02.173156+00 public.contracts_contractabi_abi_hash_key | 0 | public.contracts_contractabi_pkey | 0 | public.django_admin_log_pkey | 0 | public.django_celery_beat_clockedschedule_pkey | 0 | public.django_celery_beat_crontabschedule_pkey | 0 | public.django_celery_beat_intervalschedule_pkey | 0 | public.django_celery_beat_periodictask_name_key | 0 | public.django_celery_beat_periodictask_pkey | 37768 | 2025-09-02 12:01:18.409812+00 public.django_celery_beat_periodictasks_pkey | 725674 | 2025-09-02 12:03:33.450349+00 public.django_celery_beat_solar_event_latitude_longitude_ba64999a_uniq | 0 | public.django_celery_beat_solarschedule_pkey | 0 | public.django_content_type_app_label_model_76bd3d3b_uniq | 0 | public.django_content_type_pkey | 0 | public.django_migrations_pkey | 0 | public.django_session_pkey | 0 | public.django_site_domain_a2e37b91_uniq | 0 | public.django_site_pkey | 0 | public.history_chain_pkey | 0 | public.history_erc20transfer_pkey | 18156258 | 2025-09-02 12:01:39.042819+00 public.history_erc721transfer_pkey | 41346 | 2025-09-02 12:01:55.138353+00 public.history_ethereumblock_block_hash_key | 910687 | 2025-09-02 12:03:30.46885+00 public.history_ethereumblock_parent_hash_key | 264853 | 2025-09-02 12:03:30.46885+00 public.history_ethereumblock_pkey | 7870064 | 2025-09-02 12:03:30.46885+00 public.history_ethereumtx_pkey | 111150995 | 2025-09-02 12:03:31.747699+00 public.history_indexingstatus_pkey | 0 | public.history_internaltx_pkey | 22273831 | 2025-09-02 12:03:31.747699+00 public.history_internaltxdecoded_pkey | 13171139 | 2025-09-02 12:01:58.720673+00 public.history_moduletransaction_pkey | 619344 | 2025-09-02 12:03:00.387948+00 public.history_multisigconfirmation_pkey | 0 | public.history_multisigtransaction_pkey | 28049 | 2025-09-02 11:59:17.234762+00 public.history_proxyfactory_pkey | 0 | public.history_safecontract_pkey | 502806 | 2025-09-02 12:03:32.269589+00 public.history_safecontractdelegate_pkey | 5 | 2025-08-29 15:24:05.140483+00 public.history_safelaststatus_internal_tx_id_key | 0 | public.history_safelaststatus_pkey | 545289 | 2025-09-02 12:03:24.177223+00 public.history_safemastercopy_pkey | 0 | public.history_saferelevanttran_ethereum_tx_id_safe_096ba90b_uniq | 36851398 | 2025-09-02 12:01:55.138353+00 public.history_saferelevanttransaction_pkey | 36219003 | 2025-09-02 12:01:55.138353+00 public.history_safestatus_pkey | 0 | public.history_ethereumblock_timestamp_85f398f6 | 9 | 2025-09-01 14:30:40.369796+00 public.safe_messages_safemessage_pkey | 167 | 2025-09-02 11:09:30.379062+00 public.safe_messages_safemessageconfirmation_pkey | 0 | public.tokens_token_pkey | 7322180 | 2025-09-02 12:03:27.154744+00 public.tokens_tokenlist_pkey | 0 | public.tokens_tokenlist_url_key | 0 | public.tokens_tokennotvalid_pkey | 0 | public.unique_erc20_transfer_index | 18156339 | 2025-09-02 12:01:39.042819+00 public.unique_erc721_transfer_index | 71184 | 2025-09-02 12:03:00.387948+00 public.unique_internal_tx_trace_address | 37437 | 2025-09-02 12:01:30.394091+00 public.unique_multisig_transaction_owner_confirmation | 17737 | 2025-09-02 10:31:33.918427+00 public.unique_safe_contract_delegate_delegator | 1275 | 2025-09-02 12:02:58.228556+00 public.unique_safe_message_confirmation_owner | 1036 | 2025-09-02 10:51:27.790875+00 public.unique_safe_operation_owner_confirmation | 0 | public.unique_safe_tx_address_status | 9877 | 2025-08-30 00:20:03.333016+00 public.account_abs_sender_663a75_idx | 0 | public.account_abstraction_safeop_safe_operation_id_0bd247f6 | 0 | public.account_abstraction_safeop_signature_type_43748fb5 | 0 | public.account_abstraction_safeoperation_module_address_3646c35b | 0 | public.account_abstraction_useroperation_entry_point_2fc2134f | 0 | public.account_abstraction_useroperation_ethereum_tx_id_fd4ae6ec | 0 | public.account_abstraction_useroperation_paymaster_751e7bc1 | 0 | public.account_abstraction_useroperation_sender_d9a3df38 | 0 | public.auth_group_name_a6ea08ec_like | 0 | public.auth_group_permissions_group_id_b120cbf9 | 0 | public.auth_group_permissions_permission_id_84c5c92e | 0 | public.auth_permission_content_type_id_2f476e4b | 0 | public.auth_user_groups_group_id_97559544 | 0 | public.auth_user_groups_user_id_6a12ed8b | 0 | public.auth_user_user_permissions_permission_id_1fbb5f2c | 0 | public.auth_user_user_permissions_user_id_a95ead1b | 0 | public.auth_user_username_6821ab7c_like | 0 | public.authtoken_token_key_10f0b77e_like | 0 | public.contracts_contract_contract_abi_id_1c37b4bd | 0 | public.django_admin_log_content_type_id_c4bce8eb | 0 | public.django_admin_log_user_id_c564eba6 | 0 | public.django_celery_beat_periodictask_clocked_id_47a69f82 | 1803 | 2025-09-02 12:00:48.30518+00 public.django_celery_beat_periodictask_crontab_id_d3cba168 | 0 | public.django_celery_beat_periodictask_interval_id_a8ca27da | 0 | public.django_celery_beat_periodictask_name_265a36b7_like | 18895 | 2025-09-02 12:01:18.409812+00 public.django_celery_beat_periodictask_solar_id_a87ce72c | 0 | public.django_session_expire_date_a5c62663 | 0 | public.django_session_session_key_c0390e0f_like | 0 | public.django_site_domain_a2e37b91_like | 0 | public.history_int__from_2301f6_idx | 613554 | 2025-09-02 12:01:58.720673+00 public.history_erc20transfer_ethereum_tx_id_16a6036b | 40014 | 2025-09-02 12:03:00.387948+00 public.history_erc20transfer_timestamp_b14c5ef4 | 128 | 2025-08-29 23:45:04.175139+00 public.history_erc721transfer_ethereum_tx_id_aa020110 | 0 | public.history_erc721transfer_timestamp_aaadcf46 | 448 | 2025-08-29 23:45:04.175139+00 public.history_erc__from_64986c_idx | 470706 | 2025-08-30 00:33:16.130365+00 public.history_erc__from_72fb41_idx | 470706 | 2025-08-30 00:33:16.130365+00 public.history_erc__from_d88198_idx | 279180 | 2025-09-02 12:03:32.269589+00 public.history_erc__from_d919fc_idx | 18260 | 2025-09-02 12:02:47.611235+00 public.history_erc_address_94cee3_idx | 0 | public.history_erc_address_dba64d_idx | 88 | 2025-09-02 11:09:35.18255+00 public.history_erc_to_02d4ab_idx | 470706 | 2025-08-30 00:33:16.130365+00 public.history_erc_to_246e8e_idx | 19420 | 2025-09-02 12:02:47.611235+00 public.history_erc_to_d39948_idx | 281169 | 2025-09-02 12:03:32.269589+00 public.history_erc_to_f32154_idx | 470706 | 2025-08-30 00:33:16.130365+00 public.history_ethereumtx__from_e36d4662 | 0 | public.history_ethereumtx_block_id_92e7f70e | 22019 | 2025-09-02 12:03:24.177223+00 public.history_ethereumtx_status_d1d0de2c | 0 | public.history_ethereumtx_to_9fb37c13 | 0 | public.history_indexingstatus_block_number_40805e1c | 0 | public.history_int_to_e72886_idx | 0 | public.history_internal_transfer_from | 569468 | 2025-09-02 12:03:32.269589+00 public.history_internal_transfer_idx | 593382 | 2025-09-02 12:03:32.269589+00 public.history_internaltx__from_36470cfd | 451450 | 2025-09-02 12:01:30.31509+00 public.history_internaltx_block_number_9004df5f | 0 | public.history_internaltx_call_type_1f0c4296 | 0 | public.history_internaltx_contract_address_134466b3 | 495 | 2025-09-02 11:50:22.499358+00 public.history_internaltx_ethereum_tx_id_e6ac35ab | 37 | 2025-09-02 11:09:34.976081+00 public.history_internaltx_refund_address_09530831 | 0 | public.history_internaltx_timestamp_ef4d64cb | 5183 | 2025-08-29 23:45:05.90705+00 public.history_internaltx_tx_type_bb97604e | 0 | public.history_internaltx_value_idx | 0 | public.history_internaltxdecoded_function_name_333ae03e | 0 | public.history_internaltxdecoded_function_name_333ae03e_like | 0 | public.history_moduletransaction_module_5bf50dd1 | 0 | public.history_moduletransaction_safe | 5039 | 2025-09-02 12:03:00.387948+00 public.history_moduletransaction_to_041e4c7b | 0 | public.history_multisigconfirmation_ethereum_tx_id_71e0ab73 | 0 | public.history_multisigconfirmation_multisig_transaction_hash_4c8cbf7a | 0 | public.history_multisigconfirmation_multisig_transaction_id_43ec94d5 | 88435 | 2025-09-02 12:03:24.177223+00 public.history_multisigconfirmation_signature_type_9679d821 | 0 | public.history_multisigtransaction_ethereum_tx_id_c8d3f1cf | 39528 | 2025-09-02 12:03:24.177223+00 public.history_multisigtransaction_failed_4842ad6b | 0 | public.history_multisigtransaction_nonce_a98eecaa | 7312 | 2025-09-02 12:03:20.596274+00 public.history_multisigtransaction_safe_ba8bae68 | 101805 | 2025-09-02 12:03:32.269589+00 public.history_multisigtransaction_to_e544fdbe | 535 | 2025-09-02 12:00:02.161488+00 public.history_multisigtransaction_trusted_f8ea7938 | 6 | 2025-09-02 00:00:04.810105+00 public.history_multisigtx_safe_sorted | 20409 | 2025-09-02 12:03:24.177223+00 public.history_proxyfactory_tx_block_number_51be6689 | 0 | public.history_saf_address_aa71bd_idx | 153147 | 2025-09-02 12:03:24.177223+00 public.history_saf_enabled_020fea_gin | 0 | public.history_saf_owners_5fc97e_gin | 2097854 | 2025-09-02 12:03:33.649463+00 public.history_saf_safe_3768a5_idx | 2459 | 2025-09-02 12:03:00.387948+00 public.history_safe_banned_idx | 8075 | 2025-08-30 00:20:03.333016+00 public.history_safecontract_created_a7d29917 | 54387 | 2025-09-02 12:02:49.181985+00 public.history_safecontract_ethereum_tx_id_4f63f583 | 0 | public.history_safecontractdelegate_delegate_440a3b28 | 7 | 2025-08-31 23:05:05.029072+00 public.history_safecontractdelegate_delegator_9fd00422 | 479 | 2025-09-02 10:31:33.918427+00 public.history_safecontractdelegate_expiry_date_18014ead | 146 | 2025-09-02 12:00:05.535528+00 public.history_safecontractdelegate_safe_contract_id_4389cdbf | 0 | public.history_safemastercopy_tx_block_number_c1b3ce4f | 48231 | 2025-09-02 12:01:23.483976+00 public.history_saferelevanttransaction_ethereum_tx_id_0b704cf3 | 0 | public.history_safestatus_address_68cd154d | 0 | public.safe_messages_safemessage_safe_fc7e3fe6 | 4283 | 2025-09-02 12:01:06.401473+00 public.safe_messages_safemessageconfirmation_owner_b930d28f | 0 | public.safe_messages_safemessageconfirmation_safe_message_id_1796dddc | 0 | public.safe_messages_safemessageconfirmation_signature_type_34a29b90 | 0 | public.token_events_bugged_idx | 0 | public.token_spam_idx | 187 | 2025-09-02 11:46:28.65635+00 public.token_trusted_idx | 63 | 2025-09-02 09:34:55.595333+00 public.tokens_token_decimals_7c745c31 | 48 | 2025-09-02 11:09:36.29155+00 public.tokens_tokenlist_url_09b6a089_like | 0 | public.history_decoded_processed_idx | 117006765 | 2025-09-02 12:01:32.768036+00 public.history_decoded_not_proc_idx | 0 | (179 rows) ```

Add a test Add on redis before count as processed. Fix expiry is not setted after first add Refactor loop

moisses89 · 2025-09-08T10:15:23Z

Don't merge, this branch is at the same level as develop for testing.

Uxio0 · 2025-09-24T09:58:46Z

Is this ready to review?

moisses89 force-pushed the batch_process_safes branch from 2f652f0 to 969f16c Compare August 19, 2025 07:57

falvaradorodriguez approved these changes Aug 21, 2025

View reviewed changes

Uxio0 force-pushed the batch_process_safes branch from 5febc2c to 74922d8 Compare August 25, 2025 12:05

Uxio0 requested changes Aug 29, 2025

View reviewed changes

safe_transaction_service/history/models.py Outdated Show resolved Hide resolved

moisses89 closed this Sep 1, 2025

github-actions bot locked and limited conversation to collaborators Sep 1, 2025

falvaradorodriguez deleted the batch_process_safes branch September 2, 2025 14:49

moisses89 restored the batch_process_safes branch September 4, 2025 11:37

moisses89 reopened this Sep 4, 2025

Paginate Safes pending to be processed

7d3ed00

Add a test Add on redis before count as processed. Fix expiry is not setted after first add Refactor loop

moisses89 force-pushed the batch_process_safes branch from 74922d8 to 7d3ed00 Compare September 8, 2025 10:10

moisses89 added 6 commits September 8, 2025 15:15

Remove clear redis key for processed Safes

263c9a5

Share key for all procesing tests

abe1288

Add iterator with internal id pagination

a24c824

Fix memory consumption issue

d4763c2

Merge branch 'main' into batch_process_safes

beaf2e2

Clean redis cache when the task finalize

ac85087

moisses89 requested a review from Uxio0 September 11, 2025 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Paginate Safes pending to be processed #2616

Paginate Safes pending to be processed #2616

Uh oh!

moisses89 commented Aug 18, 2025

Uh oh!

Uh oh!

moisses89 commented Sep 8, 2025

Uh oh!

Uxio0 commented Sep 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Paginate Safes pending to be processed #2616

Are you sure you want to change the base?

Paginate Safes pending to be processed #2616

Uh oh!

Conversation

moisses89 commented Aug 18, 2025

Description

Uh oh!

Uh oh!

moisses89 commented Sep 8, 2025

Uh oh!

Uxio0 commented Sep 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants