Data Science Time Warp Machine

February 23rd, 2025 – 6:51 pm
Categorized as Computing Notes

Fedora 38 freezes up and crashes sometimes when using Gnome on bare metal.  This may be the result of Gnome reliability issues.  In a previous article I detailed creating a massive repo of Fedora 38, and I still have it.  I will not delete the 238GB repo because Fedora 40 is the last one with Python 2.7 in the repositories.  They elected to completely remove it in Fedora 41 and beyond.  I created some software in Python 2.7 that may never make it to Python 3 because I will be an old man by the time I could complete the conversion relative to my available time in the present day. I had migrated from bare metal to WSL with Fedora 36 a few years ago. I had created my own WSL instance using the Fedora 36 cloud init image, and then upgraded it over the years to Fedora 38 and then ceased updating it.  WSL crashes and cannot be relied upon to run tasks that require many hours of continuous processing.

WSL really was wonderful for development and running Linux applications with underlying Linux features.  I used it for development using Pycharm.  The problem is that I would often return after 12 hours and see a message that the terminal could be closed with a CTRL + D which indicated that the service had stopped for some reason.  I suspect these occurred when available RAM conflicted with the /dev/share features of Linux.  Troubleshooting it would take too long. I don’t trust the releases from the Windows store because forced updates in Windows can take features away or cause unexpected problems.  I upgraded my Windows 11 home desktop to Windows 11 Pro specifically so I could disable Windows automatic updates via group policies, service disablement, and registry modifications that fail to stop auto updates on Windows 11 Home.

To create a long use time capsule of sorts, I decided to switch to Alma Linux 8 from Fedora 38.  Alma Linux 9 follows the tradition of RHEL 9 and removes the easy support for Python 2.

I setup Alma Linux 8.10 Cerulean Leopard, installed from the KDE live DVD, and installed r Studio server to access via web browser.

edit /ect/dnf/dnf.conf and add keepcache=True

dnf install epel-release    
dnf config-manager -enable powertools    
dnf install R    
dnf install python2

The python2 install installs pip2.7 automatically. One calls pip2 via the pip2.7 command.

As regular user the following is required for a script I made because parsedatetime changed after version 2.5 and is no longer compatible with the previous versions.

pip2.7 install parsedatetime==2.5 --user

• Install rstudio-2024.12.0+467-1.rpm from direct download

• Install rstudio-server-rhel-2024.12.0-467.rpm from direct download

systemctl enable rstudio-server

Configure the firewall to allow 8787.

usermod -a -G rstudio-server <username> 
setenforce 0

The last instruction to turn off SELinux is temporary until I can ascertain the specific rules that will need modification to allow it work. With SELinux enforcing with the initial configuration, the server cannot be accessed via web browser remotely

Blog code and general code release note system

February 22nd, 2025 – 8:19 pm
Categorized as Computing Notes

Today’s site updates including removing the RSS feed links. Kinsta provided source code.1  The code for functions.php to remove it is:

remove_action( 'wp_head', 'feed_links_extra', 3 );
remove_action( 'wp_head', 'feed_links', 2 );

Kinsta also provided another very useful function2 and that was one that removed the various update notifications in the dashboard.

function kinsta_hide_update_nag() {
remove_action( 'admin_notices', 'update_nag', 3 );
}

add_action('admin_menu','kinsta_hide_update_nag');

Another very useful function for WordPress is one where you may add additional mime types for upload to the library.  This is useful for uploading compressed archives in different formats. Chris Meller’s venerable blog provides the source.3

add_filter('upload_mimes', 'custom_upload_mimes');
function custom_upload_mimes ( $existing_mimes=array() ) {

	// add your ext => mime to the array
	$existing_mimes['xz'] = 'application/x-xz';
        $existing_mimes['zip'] = 'application/zip';
	$existing_mimes['xml'] = 'application/xml';

	// add as many as you like
	// and return the new full result
	return $existing_mimes;

}

Somewhere along the way I picked up the following items to use as release notes within the scripts that I create.

E.g. 11/05/2024: [*] Changed URL to 1.1.1.1 from 8.8.8.8

[+] = Added
[*] = Changed  
[^] = Moved  
[=] = No Changes  
[x] = Deleted  
[!] = Bugs  
[_] = To Do  
[>] = Migrated  
[<] = Migrated

1. Kinsta®. “WordPress Disable RSS Feed,” August 30, 2016. https://kinsta.com/knowledgebase/wordpress-disable-rss-feed/.

2. Kinsta®. “How To Disable WordPress Update Notifications (Plugin or Code),” July 4, 2022. https://kinsta.com/knowledgebase/disable-wordpress-update-notification/.

3. Meller, Chris. “Modifying Allowed Upload Types in WordPress | Chris Meller,” July 26, 2007. https://blog.chrismeller.com/modifying-allowed-upload-types-in-wordpress.

The liberal eight, plus two

A ranking of counties is a serious undertaking with many different paths that one might embark upon to complete the work. The output could be considered software. Software design works best when one has a metaphor. The metaphor in this one is that of a parent deciding where to move in one of the following scenarios 1) around their child’s sophomore year in high school, 2) just after the child graduates high school, and 3) later in life when the child wants to attend college or work in their twenties. The reasons to do this vary, but generally can be summed up as ensuring a paid for college option if so desired, ensuring an intelligent pool of potential mates, offering a potential source of fulfillment for the parent as they age. Moving somewhere becomes a strategic matter with regards to wildly different labor markets and costs.

The funding of college should be the prime factor in deciding where to move. Many such programs have funding that might cease to be available in the future. Children might not want to go to college. The reason for this as the factor in determining the states is because it demonstrates a commitment to equal protection under the law. Universal higher education funding demonstrates it by offering the programs to all students and without regard for program preferences. Universal higher education funding demonstrates that the people of the state care about the young. That is not to say the states have everything in order. Tennessee for example is a well known for privation in certain segments of its economy economy, as is North Carolina. Montana has high suicide rates. New Jersey makes people think of Ben Affleck.

These states were selected because they offer universal higher education funding without regarding to the specific program selected. This selection is a subset of states that guarantee higher education funding, however many states require specific programs. These programs often purport to be high demand areas, yet the labor environment disagrees. For Example, Kentucky claims that information technology is a high demand career field, yet there are areas with a quarter of a million people in them, where one would be lucky to find five job postings for a systems job per year. There are more system administrators in one large city in Pennsylvania than the entire state of Kentucky, yet thousands of Kentucky students are told that information technology is a high demand career within the state. This is but one example. The same issue will arise in other states.

New York offers a program, but is not in the list because it incentivizes sociopathy. Wisconsin offers a program, but its continuation is determined on a per year basis and the state seems to be a contributor to the national schizophrenia. Maine and Connecticut have programs that match the criteria but they eliminated religious exemptions to vaccination. No one who loves their children should move to place that will not allow one to opt out of placing the child into a premature death lottery without the possibility of exemption for reasons of morality.

Kentucky is here due to a confluence of factors that make it uniquely interesting. E.g. one of the nation’s lowest Gini Coefficients, poor social indicators of health, labor market issues, long time spans of overwhelmingly one-party rule, with each party having taken decade long turns at the wheel, a unique position as the only state where public sector employees outnumber private sector employees, etc. In a way it is the most communistic if one considers communistic to mean that the government controls the means of production – as in the government furnishing the voter with a paycheck. That was as of my last research on the subject, but more detail will appear on the subject later.

North Dakota is here because it is apropos of the maxim “If a man wants a job, he can have one”. I lived there. My favorite economics professor is there. It’s a good place if you will risk frostbite in less 5 minutes at 50 degrees below Fahrenheit in order to live somewhere apropos of the mentioned maxim. They also have good fields and excellent bird watching opportunities.

This project will take a long time. It is a labor of love largely for myself, and one which others may take pleasure from. It may also provide useful intelligence for political activism and policy debate. Massachusetts is very costly. One of the interesting things to ascertain is whether people from less affluent places could even manage to move there in the first place. Michigan has a large Islamic contingent growing and that may prove to be a confounding variable when analyzing historical data. New Jersey is a land of high wages and UFOs, North Carolina is a red state with opposition to unemployment insurance. Over the first few counties, I will find some good information that transfers easily between analysis. Good data sources may vary between states. For example, many states suddenly hid all their vital records data such as births and deaths after 2021. The analysis will vary. My ultimate goals include formulas for the cost of housing, the probability of find a job in particular fields, and even an estimate of the probabilities that given job would have rest periods like 10 or 15 minute breaks in addition to a lunch break.

States very greatly. My non-maxed unemployment insurance in North Dakota 10 years ago was more than the 150% of the maximum unemployment insurance available in North Carolina or Kentucky a decade later. Kentucky mandates lunch and rest periods because of some hard work by the originators of the name redneck almost 70 years earlier. A similar situation exists with Massachusetts social benefits like unemployment insurance relative to those in some of the other states. The term liberal relates to the posture toward the education benefit. Some of the states in the list are profoundly illiberal in other areas. Merely offering an education program in no way assures the state’s counties’ greatness relative to another state’s.

The final states for the ranking of counties are Massachusetts, Michigan, Tennessee, Vermont, Georgia, Montana, North Carolina, and New Jersey, plus North Dakota and Kentucky. These are the Liberal 8 plus North Dakota and Kentucky.

The Edison method and a ranking of counties

December 17th, 2024 – 8:01 pm
Categorized as Social Science notes
Tagged as

The Edison method serves me well in life. It worked to stop smoking. It worked for other commitments. The Edison method is to simply announce what you will do let your integrity and shame avoidance drive you to accomplish the goal. Thomas Edison used the process when he announced what he was to invent in the future. With that in my mind, the name of this collection will change to “A Ranking of Counties”. This will become an aggregate of data and analysis regarding numerous counties in the United States.

One may recognize an allusion to “A Ranking of States” by H. L. Mencken. Mencken’s ranking of states was a legendary work which still remains readable and informative in the present day. The ranking of counties will have a similar aim albeit with a slightly different focus. The mission is this case is to function as a resource for trans-generational migration. Trans-generational migration means relocation for the betterment of one’s children and to help ensure future generations survivability through geographic location selection which best supports the continuance of the family line.

The analysis on where to place one’s future descendants will lead to some uncomfortable realities. Economics is called the dismal science after all. For example, the early death rates in almost every Kentucky county skyrocketed at the same time. What relationship might explain things like that? Those kinds of questions will be answered with data, hypothesis tests, probabilities, charts, and historical context.

A name will be introduced for the set of states containing the counties which will be analyzed. The name is the Liberal n where n is the number of states selected. In similar fashion to the G7 or G8 in the global environment, the Liberal 7 or Liberal 8 will refer to the states in this environment. The final number may not be 7 or 8, that is for example. There will also be a plus two, which consists of North Dakota and Kentucky. The analysis by county will consist of everything from employment and housing features to suicide probabilities and local colleges. The ranking is not the same as those rankings one sees about the best colleges or best town, but about the best place to plant descendants. Some of the data types that will likely be included will include sample data related to work and life conditions as reported by people in those counties. Readers may suggest counties of interest and I may review those counties at some point. Countering confirmation bias is an important task in any endeavor, so counties outside of the Liberal n may be studied for comparison.

Economics is the study of how humans allocate resources. In olden times the field was called political economy. The terms were split into politics and economics by university wanting to distinguish classes between trade oriented and government operations. The Bible contains numerous sections related to decision making around resources. Some articles in this collection may relate to prophecy and events because the events affect the economics of individuals. This work is about the making of decisions and not about the idolatry of wealth or politicians. It is impossible to separate morality, amorality, and immortality from decision making about resources. Therefore the work assumes a moral dimension. Some articles may analyze the the views promulgated by others because such views often play a role in individual decision making.

The next step in the ranking of counties will be the official identification of the liberal states for purposes of the project.

Monitoring network connections on Linux

November 25th, 2024 – 9:32 am
Categorized as Computing Notes
Tagged as ,

Here is a great TCP/UDP Port finder tool.   This was useful for finding out which port “cbt” was in my Iftop listing.  It was 7777, which was in use for the Unreal Engine on an Ark: Survival Evolved game server. Iftop is the best real time network monitoring tool for Linux that has graced my system to date — or so it seems until one finds Iptraf. Iptraf is another excellent tool for monitoring network traffic in real time.  The Iptraf manual is here.  Netstat wasn’t suitable for this purpose due to the use of UDP as the main protocol.  Netstat is certainly useful, but no switches that I could locate would enable the program to show the UDP traffic in real time and the associated addresses of those connected to the server.  Iptraf shows the UDP traffic in a rapidly updating scroll box.  Iftop shows the traffic in an easy to read display that includes the address and hostname of the connected system.

This list doesn’t relate to more in depth management tools. This list is for easy console monitoring of a server running such games as ARK Survival Evolved.

Blogging like it’s 2006

October 27th, 2024 – 7:54 pm
Categorized as Spiritual notes
Tagged as

I am blogging like it’s 2006 again. Countless people write about social justice and their experienced reality and they find praise and acclaim. I stifled my words for a long while because I believed that I would be even less able to find a job without stifling my words. My experience is valuable and would have served me well had it been in my possession prior to its acquisition.

Rebecca at Basics by Becca expressed her view on allowing the potential opinions of other people to dissuade one from following their own way. “Find out all the things that make your heart sing and go after them! It might be a little bit scary at first – but who cares? Don’t let other people’s opinions stop you from following what makes you happy. The more time we spend doing our passions, the easier it becomes to find our voice [1].” She further admonishes one to stop caring what others think about you. “This goes hand in hand with the last point – once you stop caring so much about what others think of you, you will start to feel a lot freer. It’s okay to make mistakes and it’s okay to be different! Embrace your authentic self and let it shine [1].” She recommends being bold, and letting go of the past.  The phrase about letting go of the past possesses great salience for me. “This one is hard but so important! Living in the past keeps us stuck and makes it impossible for us to find our voice because we are too caught up on what has already happened instead of living fully in this moment right now [1].” Other great action points including finding one’s tribe, expressing one’s feelings, and letting yourself be free. “Take off the mask and just BE YOURSELF with no apologies attached. Don’t worry so much about whether things are perfect or if people will judge you – just be yourself and don’t let anyone else define who that is. You are free to be anything you want, so take the time to figure out what makes you happy, go for it fully without apology [1]!” She says to be authentic: “Be honest about what matters most to you in life, because only by doing this can we be fully authentic. If society’s expectations are not on your list, ignore them and follow your own path[1]!”

Jennifer Hamady in Psychology Today mentioned that “Creativity is an imperative for a well-lived life [2].”

Sam Watson said “One of the advantages of writing blogs is that it can help to refresh and consolidate you thoughts on a topic. “ [3] His view coincides with my own.

Research has shown that writing about traumas improves time management over time. Dr. Pennebaker found that writing for short periods of time produce improvement, and writing for longer periods of time also produces improvements [4]. In an interview on the subject, Kim Mills asked “Do people have to write on a regular basis to see a really significant effect? Do I have to write something every day to really experience the benefits [4]?”

“Oh my God, no.” Dr. Pennebaker answered. “To give you an example, I write maybe two or three times a year when something miserable is going on. When I hear about journaling and writing every day, I just get nervous about it. I would hate that because I use writing when I’m dealing with something that is ugly, unpleasant, painful. And if life is going well, why in the world do I want to introspect that? I want to enjoy life when it’s here and when there’s bad things, writing for me helps me get through them [4].”

 

 

 

1. Becca, “18 Powerful Ways to Find Your Voice.”
2. Hamady, “The Truth About Creative Expression.”
3. Watson, “Geostatistical Modelling with R and Stan.”
4. Mills and Pennebaker, “Expressive Writing Can Help Your Mental Health.”

Install cron on Fedora 38

October 21st, 2024 – 9:38 pm
Categorized as Computing Notes
Tagged as , ,
dnf install cronie
systemctl enable crond.service
systemctl start crond.service

Building an anti-bitrot bunker

October 20th, 2024 – 12:18 am
Categorized as Computing Notes
Tagged as , , ,

When I first obtained a CompTIA A+ certification some time ago, the concept of bit rot related to what happens to software reliability as the number of updates to the system increases. Over time, software that worked in years past ceases to work as new system updates block system calls or change permissions and files that the software originally relied upon. This slow decaying of reliability was called bit rot. There are some other definitions floating around on the internet, but that is the one most relevant for me.

To prevent this and reduce the amount of time spent in sorting out the bit rot introduced to my investments by Microsoft’s proclivities, I have standardized on two operating systems for major time investments in computing that occupy my life.  Windows still has a place since I sometimes use Windows only games to play with my child.  Other than that necessity, I have built the things I rely on for use with Linux.  The two versions that I have standardized on are Debian and Fedora. Specifically Debian 12 and Fedora 38.  These are not what is used for the website, but they are the major components of my anti-bitrot infrastructure.  I am aware this may not be good security practice, but this isn’t to get me a job, this is to serve an aging man and his family reliably over time.

There are a few reasons that I selected Fedora 38 and Debian 12. Fedora 38 still has Python 2.7 in the repositories.  It was within 2 versions for upgrading from Fedora 36. Fedora 36 was what I was running in my Windows Subsystem for Linux instance, and I upgraded it to Fedora 38.   When deciding to move back to bare metal for my Linux software development and automation needs, I decided to standardize on that one.  Debian has a 32 bit version. I have both the 32-bit and 64-bit deployed in my network.  Debian 32 bit allows on to easily 32-bit builds of Java on a Linux server.  One can add the testing repository and have the latest Java in 32 bit form.  32-bit Java is necessary to run older Minecraft versions.  My family has a large set of mod-collections and old Java Minecraft instances and maps going back about 8 years. I can then run the latest JVM and the latest Minecraft on the same server because the 32bit JVM is available.  It is very annoying to try and  manage 32bit and 64bit Java virtual machines on the same host, so having it all 32-bit solves a huge problem.  One can add the Debian 11 repository and install Python2.7 if one wants to use old Python versions.  I need this old Python version for a project that I worked on over the course of the last 8 years.

The general anti-bit rot measure for Debian is to always use apt-get to install packages as this will leave the .deb file in the cache.  Then copy those .deb files on a regular basis to another location for use as a repository for other Debian installations.  This can be configured to work via a cron job.  Debian is really the only long-term viable game in town if one wants a 32-bit anti-bitrot bunker that will last into the future, over say, a ten year time horizon. Save the .deb files on a private web-server inside the network and periodically update that repository with the files copied via the sweeps from the cache directory.

For Fedora, this should work with any version.  Fedora 38 is the one I use.  Edit the /etc/dnf/dnf.conf file to say keep_cache=True, and this will save the dowloaded RPMS.  One can then build their repository using only what they need if so desired.  The other genuine long term standardization option is to mirror the entire repositories to the private webserver within one’s network.  To do this, install yumdownloader.   Then, move all of the .repo files from /etc/yum.repos.d except for one to a temporary location.  Then, go into a directory with a lot of space.  For Fedora 38, the complete mirror was 229GB.  Then use yumdownloader * and it will download all files from that repository.  Then, use yumdownloader –source * and it will download all source packages for the packages that it just downloaded.  Then, go back to /etc/yum.repos.d and switch that .repo file out with one that was moved previously, and repeat the process until all repositories have been completely downloaded.  Then, copy all downloaded RPMS to one large directory on the webserver and use createrepo to create the meta data.  Then on the client machines, create a .repo pointing to your own webserver, and move the existing .repo files in /etc/yum.repos.d to an archival location.   Then all of your installations will occur from your own webserver and all machines will have the same versions of packages.

In my case, I then install the data science specifics that I need for the automated software that I created.  The process varies slightly depending on whether the system is Fedora or Debian.

For Debian:

Add bullseye to sources.list
Install python2 via bullseye repositories
Install pip via the downloaded file from https://linuxhint.com/install-pip-on-debian-11/ that is to say,
1. wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
2. python2 ./get-pip.py
Apt-get install libcurl4-openssl-dev

R:
Install.packages(‘curl’)
Install.packages(‘fpp2’)
Install.packages(‘magrittr’)
Install.packages(‘urca’)
Install.packages(‘vars’)
Install.packages(‘psych’)
Apt-get install r-cran-rjava
Install.packages(‘rJava’)
Install.packages(‘xlsx’)
Install.packages(‘Hmisc’)
Install.packages(‘prophet’)
Install.packages(‘dplyr’)
pip2 install parsedatetime==2.5
apt-get install awscli
pip2 install boto3
apt-get install r-cran-car

Troubleshooting steps if .xlsx and others are not built:
Are all R packages installed successfully with a 0 exit status?

On Fedora, the repositories do not contain R components in the same way the Debian repositories do.  Here is the process for Fedora 38.

Used the script https://bootstrap.pypa.io/pip/2.7/get-pip.py
pip2 install parsedatetime==2.5 –user
dnf install libcurl
dnf install libcurl-devel
dnf install R
dnf install awscli
pip2 install boto3 –user
dnf install cmake

Within R:
install.packages(‘car’)
install.packages(‘curl’)
install.packages(‘fpp2′)
install.packages(‘magrittr’)
install.packages(‘urca’)
install.packages(‘vars’)
install.packages(‘psych’)
install.packages(‘rJava’)
install.packages(‘xlsx’)
install.packages(‘Hmisc’)
install.packages(‘prophet’)
install.packages(‘dplyr’)