Report log

10.05.2024

TCPA

Completed tasks:
- Checked why AI chat is not working.
- Made a form on the single client page.
In progress:
- Complete the integration of the new server.

Completed tasks:
- Work on exporting PDF statistics, Added the edits that were discussed. Fixed the file type.
- Help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors. No errors found
-- daily checking of the database - importing files, how and what records were added
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly. No errors
In progress:
- Script for adding records for users from admin accounts. So far, the work is only on adding simple records (without additional filters) from premium sheets

25.04.2024

TCPA

Completed tasks:
- Synchronized tcpa api servers.
- Processed state files and imported them (7 states).
- Fixed calculation of the scrub price in the modal window.
- Fixed WHM bug.
In progress:
- Complete the integration of the new server.

Completed tasks:
- Added a new version for exporting PDF statistics. There are some parts that still need to be tweaked
- Help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors
-- daily checking of the database - importing files, how and what records were added
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly. No errors
In progress:
- Work on exporting PDF with statistics

19.04.2024

TCPA

Completed tasks:
- Conducted and tested federal dnc import.
In progress:
- Complete the integration of the new email validation server.

Completed tasks:
- Received new invalid emails and excluded them from the database
- Help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors
-- daily checking of the database - importing files, how and what records were added
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly. No errors
In progress:
- Work on exporting PDF with statistics

11.04.2024

TCPA

Completed tasks:
- Rewrote state imports. Takes up less space in the database, simpler code, faster.
- Fixed a bug in importing states. CSV divided lines by \r\n and therefore read only the first line when the lines were divided by \n.
- Re-imported plates with LA state.
- Optimized federal dnc import.
- Added an error call when an sql query fails.
In progress:
- Complete the integration of the new server.
- Test federal dnc import.

Completed tasks:
- Help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors. There are still some bugs
-- daily check of the database - import of files, how and what records were added, no errors
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly. No errors
- An error occurred when displaying Price Preset and Example record for one provider sheet on the /data/ page - found and fixed
- excluded some more emails at the validation stage, based on data from the validation API
- added PDF report export for production. I uploaded a simplified version, because the full version is not ready yet.
In progress:
- PDF report

04.04.2024

TCPA

Completed tasks:
- I checked if it was possible to remove numbers somewhere directly from the base. Removed the ability to delete litigators list and serial litigators in the table.
- Optimized the state update request when importing federal dnc.
- Working on restoring the state table when import failed and restoring replications.
- Temporarily removed automatic import so that the database does not stop working.
- Added api faker for the new mail server. Added tests for it. Added global and local cache (optimization).
In progress:
- Complete the integration of the new server.
- Optimize federal dnc import.

Completed tasks:
- Help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors. There are still some bugs
-- daily check of the database - import of files, how and what records were added, no errors
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly. No errors
- Work on the server. Added fixes for environment variables. I also noticed that the optimization settings were lost. Haven't touched them yet
In progress:
- Work on exporting PDF with statistics, will be added to production the day after tomorrow

27.03.2024

TCPA

Completed tasks:
- Adapted the import of state sheets.
- Adapted the import of federal sheets to combine new numbers with state numbers.
- Added partitioning for each state (each state has a separate table).
- Adapted the import so that the partitioning is preserved.
In progress:
- Complete the integration of the new server.
- Check if it is possible to remove numbers somewhere directly from the base.

Completed tasks:
- Help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors. There are still some bugs
-- daily check of the database - import of files, how and what records were added, no errors
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly. No errors
- Added a fix for the database and logging. Some errors seem to have gone away. But for now "client denied" remains. I'll look at its logs and fix tomorrow night and reboot the server
- Added corrections to the dashboard of the sheets you wrote about
- Added a fix for the finance dashboard. Found another discrepancy in numbers last week. It did not affect the final results, but could be misleading
In progress:
- Wrote some part of exporting data from the financial dashboard. I'll try to finish it this week

20.03.2024

TCPA

Completed tasks:
- I added an API job key to the jobs on the front so that jobs on the site could be linked with jobs at the API level. Needed for debugging + may come in handy in the future if you need to do something with a job in a parallel process.
- Debug of incorrect operation of the scrub.
- Fixed saving scrub error. Due to the JSON field limit, the closing } character was not added. An error trace was saved that was too large.
- Added error messages. File processing took over $timeout minutes. Probably your file is too big or there is error in file; Insufficient credits;
- Added the text Phone verification to the scrub against section.
- Working on the integration of a new mail validation server. Partially wrote the tests. Added cache.
In progress:
- Adapt the import of state sheets.
- Adapt the import of federal sheets to combine new numbers with state numbers.
- Complete the integration of the new server.

Completed tasks:
- Help Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors. There are still some bugs
-- daily check of the database - import of files, how and what records were added, no errors
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly. No errors
- Added fixes for importing records, tested for different scenarios.
- Added fixes to the finance dashboard, the errors of which I found last week
In progress:
- I worked a little on the server and, it seems, found errors that I reported earlier. But I didn’t have time to add and test, so I left it closer to the end of this week, when the site would not be overloaded

14.03.2024

TCPA

Completed tasks:
- Added removal of temporary files.
- Michael's files. For this task, a data filtering script was written + auto-detection of columns with numbers was added in the duplicate removal script.
- Updated API documentation.
- Updated state statistics in the modal window.
- Added a cron that updates the states for numbers without states. + Added to wiki.
- deleted the numbers without states, having made a copy before.
- Fixed the display of graphs on the API statistics page.
In progress:
- Adapt the import of state sheets.
- Adapt the import of federal sheets to combine new numbers with state numbers.
- Complete the integration of the new server.

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
-- daily checking of all purchases for errors, bugs, credit usage, etc. No errors found
-- daily checking of server logs and vp for errors. There is an error in server configuration. I had the same one on the TCPA, I’ll write to Nariman, since he already corrected it for the TCPA
-- daily checking of the database - importing files, how and what records were added. Of the bugs I worked on last week, there is one. I'll write about it below
-- Once every three days, after checking the numbers for the presence of TCPA, I selectively check some numbers through the TCPA website to make sure that the API is processing correctly
- Added the ability to publish/hide sheets on the sheets dashboard. I left the idea of doing the same for the sheet sticks, since you can only make 5 sheets of sticks - the roll of sticks will confuse the sticks more than it will help
- backup the table of records from the database and clear invalid emails
In progress:
- I mentioned this error earlier. Some emails, names, and other columns can be filled with empty values instead of NULL. This error was a long time ago, but... it did not greatly influence purchases and imports; it was not a priority. But after checking the database using our APIs, such “” (empty) values began to interfere and sometimes display incorrect data. A fix has been written for some of them, but has not yet had time to test it properly for production

06.03.2024

Completed tasks:
- Added aliases for tcpa api scrub types.
- Wrote an email validation server in go lang to increase speed. Added cache.
- Wrote a benchmark for a new server. The comparison was posted on github https://github.com/Cashyew/supportdatabases.com/issues/1.
- Started integrating a new server for use. To do this, I installed the golang environment on the server, configured the supervisor and started adding code, but I haven’t finished yet.
- I installed a proxy on the server so that I could test the validation locally.
In progress:
- Adapt the import of state sheets.
- Adapt the import of federal sheets to combine new numbers with state numbers.
- Check deletion of temporary files.
- Complete the integration of the new server.

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors.
- on the finance page of the dashboard - changes to the filter of the 3rd section and use of the filter of the second section
In progress:
- Records Lists mass edit

29.02.2024

TCPA

Completed tasks:
- Replace default pop-up windows (title) with custom ones.
- Add state count output to the UI.
- Check woocommerce message.
- Increased block storage for data api
- Stopped file processing on the data api if there was less than 1GB of disk space left
- Checked and updated tcpa api documentation
- Fixed a bug with clickhouse
- Added information to the Special State Mini TCPA Laws page
- Fixed the long loading time for the admin panel on tcpa (questionable, but no complaints yet)
In progress:
- Adapt the import of state sheets.
- Adapt the import of federal sheets to combine new numbers with state numbers.
- Check deletion of temporary files.

Completed tasks:
- viewing logs, purchases, database, etc., searching for errors. There were many requests from Michael and Harvey. I spent almost all my time working on tickets and requests.
- One of the tickets was about an incorrect price - when selecting records on the /data/ page, the price was one, and when purchasing on the /checkout/ page, the price was different. Found and fixed the error.
In progress:
- worked a little on the finance page of the dashboard - changes to the 3rd column filter and use of the second column filter

22.02.2024

TCPA

Completed tasks:
- Review and release state counts.
- Added dynamic pricing for filtering by state. Changed the old price settings to new ones.
- I added a meta field to the scrub API that indicates the price of the scrub.
- The scrub on the site takes the price from the API response.
- Moved the state selector.
- When an API request is made by an admin, NRA is used instead of NRA_API. It is necessary that the price on the website matches.
- Fixed autopull.
- Added display of free space on the system page.
- Fixed the icon on the modal window.
- Optimized site speed (configured opcache, installed and connected redis).
- Added buttons to the state selector.
In progress:
- Replace the default pop-up windows (title) with others.
- Adapt the import of state sheets.
- Adapt the import of federal sheets to combine new numbers with state numbers.
- Add state count output to the UI.
- PHP optimization.
- Check deletion of temporary files.
- Check woocommerce message.

Completed tasks:
- viewing logs, purchases, database, etc., searching for errors
- fixed bugs on the cashew side. errors that occurred:
-- there were many emails in the database that contained spaces and new lines both in front of the email and at the end. When sending such emails and receiving them from the API, a problem arose - they were already added without spaces to the statistics, after which they could be sent again. cleared all emails of spaces, tabs, etc. from base to cashew. made the selection of emails and adding them to statistics more stable
-- There was an error regarding the frequency of sending data to the API. It happened that without finishing the first cron launch and receiving results from the API (frequency of 10 minutes for 2k emails, which in turn were sent at 1k to the API), the second, third, etc. began. cron launches. What, in the end, loaded the API and received response errors from the API.
-- + To the second, repeated sending of the same emails was added, since Cashew has not yet received a response from the API. And if the API managed to cope, then the same emails were sent to the statistics. As a result, a huge number of duplicate emails were included in the statistics. I added a sequence for sending requests to avoid these situations. Although because of this, one bug was discovered on the API side - the same emails were first marked on the API as valid, and after the 2nd or third sending - as not valid
-- Regarding the error with email spaces, I fixed them too. Even if they are not sent for validation, all spaces are cleared on the cashew side
- For Now about 50% of emails have been validated through the API. I’ll let you know when the verification is complete
In progress:
- Statistics dashboard

15.02.2024

TCPA

Completed tasks:
- Add high risk states to the state map.
- Add a high risk mark next to the states in the state selector.
- Replace the state selector title with federal if no states are selected.
- Add State DNC type.
- Exclude State DNC if not selecting a state.
- Added state counting cache
- I calculated total matches based on the sum of matches for each type.
- Updated SDR and High risk icons.
- Updated documentation.
- Added 10k coins to all subscribers.
- Added counting of matches by state. (Local for now).
- I loaded the current database everywhere and fixed replication.
- I replaced the default encoding on the slaves like the base mester.
- Cleaned temporary files on the api.
In progress:
- Replace the default pop-up windows (title) with others.
- Adapt the import of state sheets.
- Adapt the import of federal sheets to combine new numbers with state numbers.
- Review and release state counts.
- Add state count output to the UI.
- PHP optimization.
- Check deletion of temporary files

Completed tasks:
- viewing logs, purchases, database, etc., searching for errors
- fix after updating plugins and themes, searching for new errors
- fix for the Revenue column. In some scenarios, Revenue was added to the database as "0"
- fix for finance dashboard
- fix for downloading the entire sheet from the admin panel and the dashboard of sheets that Michael wrote on
- closed the ability for providers to download all records from the sheets they added - until we decide what to do with this functionality
In progress:
- I wanted to provide statistics on the difference between invalid emails with Millionverifyer and SDB Validate Emails API, but the verification through our API has not yet been completed - new bugs have appeared, our API is very slow at checking (2k emails every 10 minutes, and even then it does not always have time to check), I also had to turn off checking until I figured out the bugs on the cashew side, and I also had problems when importing records for new sheets on Monday

08.02.2024

TCPA

Completed tasks:
- Wrote a script to add coins, added coins.
- Transferring federal DNC to state table.
- Added SDR marker.
- Added a page of state laws on tcpa and added a link to it.
- I added a page with a map of the states and added a link to it.
- Updated the information on the number verification page and updated the design.
- Added an update to the button for selecting federal dnc to state dnc and back, depending on the set of selected states.
- Fixed pagination in tables.
- Removed the (POST) mark next to the field name on the documentation page.
In progress:
- Add high risk states to the state map.
- Add a high risk mark next to the states in the state selector.
- Replace the default pop-up windows (title) with others.
- Replace the state selector title with federal if no states are selected.
- Replace scanning by federal with state if all states are selected.
- Adapt the import of state sheets to work by marker in the state table.
- Adapt the import of federal sheets to combine new numbers with state numbers.
- Replace federal_dnc in DNC at the scrub level on the site. Also replace States with federal if no states are selected, a list of states if not all states are selected and all states if all states are selected.

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors
- full integration with SDB email validator for all emails. statistics have also been added for emails during import
- searching for errors using Cron. Ultimately, Nariman fixed it
- work on the financial dashboard - comparison of the results that were shown with the results that were manually searched (sales from the lists of providers), another fix was added to reset the profits of providers - the error was related to the values after the decimal point in numbers
In progress:
- fix for displaying Total Orders when a provider is selected on the dashboard of sheets

02.02.2024

TCPA

Completed tasks:
- Added the ability to scrub all states. Before this there was a limit of 8 states.
- Updated state selector to work with all states.
- Scrub for all states only works if equal or more than 50 states or 0 states are selected.
- Added a list of states to the modal window with the scrub result.
- Updated documentation page, namely styles and text.
- The API has been switched to by-states mode for all requests.
- Added storage of selected scrub types, including states.
- A page with a tcpa map was created, added to the menu, and the error in displaying data on the map was corrected.
- Renamed the Samples menu to Demo.
In progress:
- Add data to ai chat.
- Add information by state

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors
- fix for sdb api, adding search only by email
- also added search by email when importing records
- fix for the finance dashboard. An error popped up when Michael made a payment to providers and it was necessary to reset the statistics
- fix when importing sheets. one of the imports went through with an error in staging; with a certain file it would have given an error in production too
- adding a warning and other parts of the task when importing date records by the record provider
- disabled cashew cache for emails. Now all emails are sent to SDB API for verification
- text fix - Michael's request

26.01.2024

TCPA

- Update API - operations by state.
- Update scrub interface - operations by state.
- Releasing States update today/tomorrow.

- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors
- fix for sdb api, adding search only by email
- also added search by email when importing records
- fix for the finance dashboard. An error popped up when Michael made a payment to providers and it was necessary to reset the statistics
- fix when importing sheets. one of the imports went through with an error in staging; with a certain file it would have given an error in production too
- adding a warning and other parts of the task when importing date records by the record provider
- disabled cashew cache for emails. Now all emails are sent to SDB API for verification
- text fix at Michael's request

18.01.2024

TCPA

Completed tasks:
- Added filtering by state for dnc and dnc fed on the TCPA API
- Added tcpa api version switching using the version field
- Added state columns to the tcpa and dnc complainers tables and filled them in
- new fixes for incompatibilities with php8+
- "telecorp" subscription fix
In progress:
- Tcpa states UI

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors
- fixing some parts of SDB Find data, adding logging/statistics, testing and launching for production across the entire database.
the entire database was checked by phone number:
records found with SDB Find data - 2,339,425.
added missing data to 2,130,717 records.
- new fixes for incompatibilities with php8+ and the new WP
In progress:
- trello tasks

CD/Data API

Completed tasks:
- Added graphs on the Analytics page of API queries on Data api
- Added deletion of unnecessary information (files and tables) for tasks that failed on the Data api
- Cancellation of payment if stripe card has a bad rating

11.01.2024

TCPA

Completed tasks:
- Plugin and WP Update
- PHP update
- Due to the update, where user files with a check mark should be deleted, the variable in which the path to the file changed (the slash was deleted) due to this, new files were not deleted from the server
after updating php in mysql, an error appeared due to which LOAD LOCAL INFILE did not work, fixed
- Added states to the tcpa table.
- Added filtering by state for tcpa and troll
In progress:
- dealing with file errors in the scrub

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors
- I found and fixed some more bugs related to the update. I added them through the file manager.
- there is a ready-made script for checking the entire database via SDB Find data, but without tests
- after one of Michael’s queries, I found many records in the database for the columns first name, last name, company, and the values of which were empty strings. I fixed them, most were old, but they were also added last year. I haven't looked for the reason yet
In progress:
- I will test it in staging and launch it via cron in production today after testing the SDB Find data of the entire database.

CD/Data API

Completed tasks:
- Fixed a bug on the CD
- Migrated from hdd storage on nvme to sdbs, transferred the database. Deleted the old storage.
- Made changes to the data api
In progress:
- Add filtering by state for dnc fed and dnc complainers
- Cancellation of payment if stripe card has a bad rating
- Add Paypal as a payment method

05.01.2024

Most of the time this and last week was spent on PHP upgrade on servers.

TCPA

Completed tasks:
- Configured clickhouse (for logs).
- Update of Wordpress and plugins.
In progress:
- PHP upgrade. Should be finished today

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors
- Update of Wordpress and plugins
- PHP upgrade.
In progress:
- Spout library config

CD/Data API

Completed tasks:
- Added logging of search requests to the data api

- Reconfigured postgresql to work on hdd
- Added an analytics page for API requests (no graphs yet)
- Added a tab with the latest running jobs (horizon for some reason does not have such information)
- Optimized the work of some jobs and analytics pages, made some fixes
In progress:
- Add graphs on the API query analytics page
- Cancellation of payment if stripe card has a bad rating
- Add Paypal as a payment method

29.12.2023

TCPA

Completed tasks:
- Letters about the expiration of a credit card and a letter about a small balance.
- Changes in deleting cleaned files. Works via ajax, added capability for users to delete files, added file completion time.
- Added an order modal for users in the users table.
- Added subscription reactivation and confirmation button.
- Scrub page Instructions update.
- Cronjobs cleanup.
- Dropbox files configuration.
In progress:
- Update of Wordpress and plugins
- PHP upgrade.

Completed tasks:
- help Michael and Harvey
- viewing logs, purchases, database, etc., searching for errors
- Update of Wordpress and plugins
In progress:
- Adding and displaying data for the SupportDatabases Find Data API on the dashboard page - system
- improving the logging system so as not to lose information
- continuation of work on "List fields"
- new (latest) fix for the incorrect operation of the map provider - the sheet created by it. Some bugs were associated with it. Example: the provider created a sheet - ID 1444, provider ID 50. In the database (ntcash_chw_list_id_provider_id) some records were saved as list_id=0, provider_id=50 or list_id=1, provider_id=50

CD/Data API

Completed tasks:
- Created and connected external storage to sdbs
- Moved the postgresql database to external storage
- Added a script for TCPA that sends files to the Data API
- Added filters for the table of uploaded files on the Data API
- Added logging of long queries to postgresql sdbs
- Lots of fixes and optimizations on the Data API
In progress:
- Add request logging to the data api (I’m thinking of trying it on elasticsearch)
- Cancellation of payment if stripe card has a bad rating (CD)
- Add Paypal as a payment method (CD)

22.12.2023

TCPA

Completed tasks:
- Released a WordPress optimization update. This update was described in detail in the git and in the logs.
- Table of numbers by state plus a map with the number of numbers.
- MailWizz sheets have been updated and added automatic addition of users to sheets. I described the trello in detail in the card.
- Page showing the transaction for users.
- Trello cards about hiding coin pages and corrections in mail.
- Added a guide to AI assistant
- Removed SalesIq and made ai chat available to everyone.
In progress:
- Reminder about the credit card expiration date.
- A reminder about 5% of the balance.

Completed tasks:
- help Michael.
- viewing logs, purchases, database, etc., searching for errors
- Added sending of imported files to the SupportDatabases API
- Added a fix when creating date providers for new sheets
- Redirected Payouts from the admin panel to the finance dashboard, so as not to catch bugs with negative payments again
- Added a fix for the modal when users try to register, log in or use the Forget password form. The fix has been added via the file manager to the UsersWP plugin. The modal was not visible, and even after the CSS fix there were bugs regarding the normal display of the modal. The fix has so far been added to the trello “What to do after the update”, because... Most likely, in some new version, the developers of this plugin will add it to the release.
In progress:
- Financial dashboard update
- WP optimization

CD/Data API

Completed tasks:
- There was an error with the backup to CD. (The pg_dump alias was rewritten to the old version)
Data API:
- Added pause and queue buttons.
- Added the ability to stop processing new downloaded files.
- Fixed some bugs.
- Optimized queries to the graph database. (Data processing froze because of them)
- Created an account specifically for CL and an API key. Samvel and I figured out how to send a file to the Data api.
In progress:
- Add request logging to data api
- Cancellation of payment if stripe card has a bad rating
- Add Paypal as a payment method
- Try to transfer the data api database to data storage

15.12.2023

TCPA

Completed tasks:
- WordPress optimization, a lot of changes released. Gtmetrix score is now about 90.
https://docs.google.com/document/d/1_p1oE7NAk3981K7V_0UBmjSPvEq9wvbWuWUaJ3PvcCs/edit
- Duplicates error has been fixed
- FAQ scrub calculations info added
In progress:
- Table of numbers by states, there is no complication here, the complication was with optimization, I wanted to finish it quickly to show it. Tomorrow or at most Thursday I will show the table, I still need to update the playlists because there are complainants to whom staff have not been assigned, but there are staffs in the Excel files.

Completed tasks:
- help Michael and Harvey.
- viewing logs, purchases, database, etc., searching for errors
- "Number of inactive lists (Lists that were previously posted but are private). - added to the dashboard
- Warning to the provider if he did not select the main phone number during mapping. When you first click “Import” a warning pops up (see first screenshot). When you click the second time, import is already in progress. Please let me know if you need to change the warning text.
- Added a fix (more of a temporary crutch) for displaying the number of records of generic sheets, Until I find a faster query to the database for all scenarios.
- Slightly adjusted the creation of temp tables in the database for Aged Generic records. Now Aged records are considered to be more than 12 months old, but the generation is for time: 12 months ago, but the first day. Example, 2022-12-01 instead of the old 2022-12-12 generation. Every day a new tempo table will not be created when viewing the Aged tab
- changed the sorting of created provider sheets on the import page - from new to old
In progress:
- Sending files to Data API fix
- Looking for suitable caching plugin. Testing different ones.
- "List fields". When importing files into a sheet, once again at a certain time (I think once a week will be enough) popular columns are added to the table (I’ll do it through the settings in the admin panel so that you can set the percentage, by default it’s more than 50% for now) for each sheet . And at the front in the filter I show only the columns that the majority (> 50% or another figure) of the records have. The rest is ready, simple filtering by records is already underway

Data API

Completed tasks:
- Add captcha to receive SMS on CD
- Improve the data api analytics page
- Hide initial message from Chat AI for non-logged in users
- There were errors and slow requests to the data API - fixed
- I set up postgres logs, it took up 7GB
- Improved uploading files to the data api, now you can download them from s3 storage (they are transferred there to save space on the server itself)
- Updated the post.sdbs certificate
In progress:
- Add request logging to data api
- Cancellation of payment if stripe card has a bad rating
- Add Paypal as a payment method
- Add a guide to AI assistant

6.12.2023

TCPA

Completed tasks:
- The verification error has been fixed.
- Michael and Harvey's requests
- WordPress optimization. Finished but not released yet.

Processing:
- Show TCPA numbers by state. A day or two, I fill out the states using excel files that were not filled out.

Completed tasks:
- help Michael and Harvey.
- viewing logs, purchases, database, etc., searching for errors
- adding a “new” icon, which was not activated immediately. The "new" icon works the same as with old sheets, except that the week is counted after activation.
- changes to the sheet queue on the sales /data/ page. Now, two columns are used to order the sheets: "created" and "activated_at". "created" is used for backwards compatibility, at the end of the week it will no longer be needed. New activated sheets will now be at the top of the sales table
- fix when importing date sheets to providers. Now they cannot select prices for one sheet more than $6
- adding a "Full name" column when importing sheets by providers. Let's see how they will use mapping: will they indicate "full name" if there is only one field with a full name in the file.
- adding statistics for this request:

We need to add to data provider statistics like
Number of lists
Number of Published Lists
Number of Private Lists
Number of inactive lists (Lists that were previously posted but are private)."

Processing:
- I found the cause of another bug in the purchase statistics of the date provider - when minus prices were shown in the statistics. But for now I added logging to be sure
- added some new logging (for now I’m only writing errors in the main log of the VP, and not in a separate folder for this api) for the Find data API
- in principle, this functionality already exists, but I haven’t looked at the results yet, so I haven’t added it - “Number of inactive lists (Lists that were previously posted but are private).”
- caught errors to prevent providers from loading records if the main number is not selected when mapping with the database. I'll fix it and add it today
- fix for displaying the number of records of generic sheets - I also had problems with it, so I didn’t have time to finish it
- sending the file that the provider imports to DataAPI. I just need to test it, then I’ll add it
I think I was working on something else last week, but I don’t remember now.

CD/Data API

Completed tasks:
- Added the name and start message for the AI assistant.
- I connected AI chat to tcpa and made it visible only to admins.
- Installed and configured backup on CD. Saved to S3 storage every 3 days.
- Added cookie consent.
- There was a horizontal scroll on the CD, I fixed it.
- I wrote an article on how to log in to the AI chat admin panel.
- Uploaded files from CL to Data api and added them to the processing queue.
- Made some fixes on the Data api side.
- Added a page with analytics.

Processing:
- Add captcha to receive SMS on CD
- Improve the data api analytics page
- Add request logging to data api
- Cancellation of payment if stripe card has a bad rating
- Add Paypal as a payment method
- Add a guide to AI assistant
- Hide initial message from Chat AI for non-logged in users

29.11.2023

TCPA

Completed tasks:
- There was a bug on the API side; if the number had a duplicate, then in this case only the duplication status was shown.
- Due to the API duplicate bug, I wrote additional functionality for the scrubber that counted duplicates and added them to the result at the end. Because of this, there was a problem with counting several columns initially. After the correction on the API side, it became much easier, now additional calculations of duplicates are not needed. Everything works as it should.
- Updated FAQ page content
In progress:
- WordPress optimization. This is one big task that is divided into small parts, more details here. https://docs.google.com/document/d/1_p1oE7NAk3981K7V_0UBmjSPvEq9wvbWuWUaJ3PvcCs/edit

Completed tasks:
- help Michael and Jeff.
- viewing logs, purchases, database, etc., searching for errors
In progress:
- Changes in click statistics over time (almost finished, I’ll do a merge for production immediately with other changes on the environment)
- adding new data to the dashboard at Michael's request

CD/Data API

Completed tasks:
- Make the data api paid
- Collecting information on sheets for optimization supreshna (whether the user has purchases, to take into account whether they need to be excluded)
- Add a filter for sheets with failed imports
- There was a validation error. It was not possible to use a space in the sheet name and description. Added rule NameWithDigitsRule, which validates names.
- There was an error with progress calculation. corrected + added test
In progress:
- Cookie consent
- Contact page security
- Fraud score

24.11.2023

TCPA

Completed tasks:
- verifonix.com. Site copy created, but scrub is not working yet, as we need weather to get one server for verifonix API, or reconfigure scripts to work with API on different domain.
- Sample scrub lock
- Team requests
In progress:
- Several columns scrub
- Speed, SEO optimizatiom

Completed tasks:
- help Michael and Harvey.
- viewing logs, purchases, database, etc., searching for errors
- MailWizz integration for Cashew. The functionality is divided into two stages:
-- After activating the plugin and adding the API case, a “bulk send” button appears for admins. From the admin panel she adds all users and users with purchases to two Miles sheets. After which the button disappears so as not to send the same users again and again.
-- adding new active users and new users integrated with WP and WC functionality via hooks (error now for new active users)
- Fixed layout for the guid functionality
- Fix for the age of new sheets, when the sheet is activated not upon creation, but after a while
In progress:
- Optimization of the site and number of requests for files
- Changes in click statistics over time

CD/Data API

Completed tasks:
- Fix mail on CL
- Collecting information on sheets for optimization supreshna (are there general records to take into account whether an exception is needed) + tests.
- Added API documentation to ai chat
- I improved the supression script so that it could find the phone speaker itself (Michael sent me 10GB of files in different formats)
- Working on Michael's files
In progress:
- Make the data api paid.
- Collecting information on sheets for optimization supreshna (whether the user has purchases, to take into account whether they need to be excluded)
- Add a filter for sheets with failed imports

15.11.2023

TCPA

Completed tasks:
- Fixed a bug with subscription credits
- Fixed a bug with redirecting after selecting a subscription
- finished with the mailwizz plugin
In progress:
- sample scrub
- verifonix.com

Completed tasks:
1. help Michael and Harvey.
2. viewing logs, purchases, database, etc., searching for errors
3. added fixes by clicks, organized them
4. Found a bug and added a fix for loading records into sheets. Mainly the problem started to arise for data providers. Some recording files threw an error, and as a result, the download automatically received the "error" status and stopped
5. added a “date providers” filter for the sheets dashboard
6. fix for errors related to the display of sheets on the sales page, Users most likely did not notice them.
In progress:
1. Adding a "Full Name" column for date providers
2. Adding a timestamp when viewing the "Premium" and "Generic" tabs on the /data/ page. I created a new database for it and the data will be added/taken from there, otherwise the old option of storing it in usermeta is no longer suitable - the table will be too cluttered
3. Test optimization plugins
4. Link dashboards together

CD/Data API

Completed tasks:
- Add revenue filter to ui
- Fix a bug with stripe payment (the refund is made to the account balance)
- Fix bug with rollback after unsuccessful purchase of a sheet
- Launch AI assistant
- Make a wp plugin to connect to AI assistant
- Fix chat (align to the right, fix height and remove "powered by chaindesk")
- Code refactoring, filter optimization, fixes on CD
- Parsed the WTO docks for Michael
In progress:
- Make the data api paid.
- Collecting information on sheets for optimization is essential (are there general records to take into account whether an exception is needed, whether the user has purchases to take into account whether they need to be excluded, etc.).
- Add a filter for sheets with failed imports

10.11.2023

TCPA

Completed tasks:
- Michael's requests, filtering files, getting numbers by matches
- Finished with the MailWizz plugin, already on saging and mails are sent through the plugin. Added logging of all requests.
- Working on tcpa 1025 replication (the database was stuck on a transaction. I uploaded a fresh database and restarted replication) + added a possible fix

In progress:
- Finished with the sample scrub form, added fields for mail and phone numbers on all sample scrub forms, all mails are sent to mailWizz automatically. Should be released till monday.

Completed tasks:
1. help Michael, Jeff and Harvey.
2. viewing logs, purchases, database, etc., searching for errors
3. added fixes to the dashboard of sheets that I launched earlier using the file manager
4. added the System page to the dashboards, where click reports are currently shown. Prepared functionality for a larger amount of data for this page. There are not many options yet, but there are thoughts. In the evening I’ll post what else you can add there
5. added fixes for importing files. found some errors. In the last week, date providers have added a lot of sheets. I fixed one of the errors - if the first entry in the file did not contain a number or email, then the entire check for this column was discarded and was not carried out
6. some Revenue were in the form of an array, did not pass the check and were not added to the database, throwing a warning. I have fixed the check, although I have not yet found the reason, I will continue to work on it.

CD/Data API

Completed tasks:
- Added revenue parser
- Added revenue columns to cider with Revenue meta type
- Added Job, which parses revenue columns + tests
- Added a property to the sheet statistics hasRevenue + tests
- Added a revenue filter (not added to ui yet) + tests
- Partially prepared the infrastructure for ai

In progress:
- Make the data api paid.
- Collecting information on sheets for optimization is essential (are there general records to take into account whether an exception is needed, whether the user has purchases to take into account whether they need to be excluded, etc.).
- Add a filter for sheets with failed imports
- Add revenue filter to ui
- Fix bug with stripe payment
- Fix bug with rollback after unsuccessful purchase of a sheet
- Launch AI assistant

02.11.2023

TCPA

Completed tasks:
- Corrected the registration page, sample scrub and pagination in the tables
- Michael's Requests

In progress:
- Plugin for MailWizz. I'm working on bulk sending users to a sheet
- Transferring a site to another server

Completed tasks:
1. help Michael and Harvey. It took approximately 2 hours.
2. viewing logs, purchases, database, etc., searching for errors
3. work on the dashboard of sheets.
4. adding Hotjar tracker

CD/Data API

Completed tasks:
- Made tests of purchases through the browser (including with the states filter)
- internal refactoring, wrote auxiliary classes, traits, etc.
- Removed the choice of the number of sheets per page
- Found ways to make our own AI agent (I figured out how it works):
https://github.com/gmpetrov/databerry/tree/main
https://www.langchain.com/
https://github.com/Ironclad/rivet

In progress:
- Make the data api paid.
- Collecting information on sheets for optimization is essential (are there general records to take into account whether an exception is needed, whether the user has purchases to take into account whether they need to be excluded, etc.).
- Revenue filter
- Add a filter for sheets with failed imports

26.10.2023

TCPA

Completed tasks:
- Fixed corrupted records in tables
- Michael and Harvey's requests
- Problems with 101 api were fixed
In progress:
- New notifications by mail to users, all that remains is to configure the text itself

Data API/CD

Completed tasks:
- Optimized price calculation for the sold count preset
- Added filtering by state (not released yet, need to write tests)
- Added stripe testing
- Installed and configured laravel dusk for testing via browser
- Made force https only for the remote server (it got in the way on LAN)
- Working on Michael's files
- Wrote a script that takes certain columns from tables and puts them in one table
In progress:
- Make the data api paid.
- Collecting information on sheets for optimization is essential (are there general records to take into account whether an exception is needed, whether the user has purchases to take into account whether they need to be excluded, etc.).
- Revenue filter
- Add a filter for sheets with failed imports

Splitting CSV file

Remove duplicates

Format TCPA list file

Find number in CSV file

Report log

No Comments