Python3 has a very neat library in it that will save you a lot of unnecessary heavy lifting processing and re-engineering your average web 2.0 json output api. It’s pretty neat how if you know what your doing not only can you implement a really cool monitoring API (not that you’d need to there are plenty of 3rd party ones available like newrelic, datadog and many others), but perhaps you may have proprietary reason to develop your own internal monitoring, and various other reasons.
Also, when it comes to monitoring server side processes, states with application layers or microservices there is more going on in the middleware, but it’s exactly the same process except in your app.get you are going to define a function that checks for output or inputs of another program, and so on, be that a datastore, text file, api call, or conditional developing API with python is made really easy. We will use uvicorn to run the python-server.
import uvicorn
import os
import psutil
from fastapi import FastAPI
We also are using the fastapi library to generate our arbitrary server calls for the client. We use os,psutil to access certain os functions for the other calls.
We might be using the API call for our own custom service tracker, trigger alarm, alert, such as if bruteforcing etc is happening maybe the trigger that consumes this alarm can set an upper limit or range for its rate_of_change and so on. This simply provides output for the number of requests made. A seperate read_only function could be made that only reads the variable by removing hits+=1.
/health
A pretty standard thing for any monitoring APi is the health endpoint, normally an “true” or “OK” statement string suffices, but some people run much more conditional health that could check an array of resources or other api endpoints values before returning the client output. I haven’t done lots of development with Python but it seems like a pretty cool language worth time, and with Flash and uvicorn a pretty compelling usercase exists for rapid prototyping and development.
I thought it would also be cool to try my hand at writing a basic disk metric. I am not sure if this was the best way to implement it and it is kind of hacky, you may not want to monitor /, but a specific path with a specific disk which is simple enough to do by altering the statement to reflect the path.
## while we're at it lets generate /memory and /cpu metrics too cause its really easy todo
@app.get("/memory")
async def memory():
# Getting % usage of virtual_memory ( 3rd field)
current_mem_used = psutil.virtual_memory()[2]
print(current_mem_used)
if current_mem_used < 90:
return "ok"
else:
return "not ok"
@app.get("/cpu")
async def cpu():
# should probably use aggregates for loadavg perhaps
current_cpu_used = psutil.cpu_percent(interval=10)
max_percent_load = 90
print(current_cpu_used)
if current_cpu_used < max_percent_load:
return "ok"
else:
return "not ok"
Example of Service Status endpoint for Nginx
Since I was at it I thought it worthwhile to spend a few minutes creating an arbitrary and hopelessly basic check of nginx using psutil. I am wondering if it is better to use systemctl status output or some other library instead, I don’t like using unnecessary libraries though unless it is making things a lot less convuluted by doing that.
# we need this helper function to get process matching for a given service as an argument
def is_process_running(name):
for process in psutil.process_iter(['name']):
if process.info['name'] == name:
return True
return False
# with the above helper function we can use this simple notation for all desired processes
@app.get("/nginx")
async def nginx():
if is_process_running('nginx'):
return "ok"
else:
return "bad"
Starting the API Service application
Much like with nodejs and scala its a pretty simple framework/lib to deploy to a server, just append this to the bottom of your python file and then execute as normal.
if __name__ == '__main__':
uvicorn.run(app, host="0.0.0.0", port=8000)
python3 monitor-api-service.py
Let’s go ahead and open up this in a browser by navigating to http://ipofserver:8000
Pretty neat. Note this is running as http:// to run as https:// you will need a certificate and to define it like
In this case because the HTTPS transport and cert are a seperate layer to the plaintext api they aren’t really anything to do with the implementing of the API itself and moreover to do with termination of the TLS certificate at the tld, in any case perhaps you want to implement properly with https. You can get free certs from letsencrypt using the certbot commandline utility.
The power of developing api rapidly like this is really eye-changing for anyone who had a complex server or rpc service backend, as perhaps they’ll see that they only need only translate the necessary command functions with such approach to provide nearly any data thru the api from any service or corresponding conditional test.
Producing something like this is easy but doing it correctly in the first place will always be the real challenge.
Greetings, for those who know me my quiet (but probably quite extensive) interest in data theory has been brewing in my latest fun-but-semi-commercial project, cosmic OS, among them generative computation, fascinating to me as a musician and also a technical engineer I’ve watched over the last 20 years what were previously far away pipe-dreams and ludicrous ideas are becoming increasingly probable computationally, as mathematical and computer modeling systems evolve substantially. Some of these concepts seem important for bio feedback and development of neural interface but it beyond the scope of this article. I will include more details about the increasingly theoretical probabilistic neural interface in my next post.
Although my technical researches and hobby interests in sub space field mechanics and imaginary and real complex planes has been a generally a private interest some of the more recent invokations of AI and ML, in particular generative adversarial networks have started to support some of my earlier ideas and conclusions about generative computation and the square root of -1 in relation to m-verse theory, quantum computation and intelligent systems. We are approaching computer systems that can explore for us, and, eventually with us in fascinating ways never conceived before. There is some engineering and computational gaps that exist in this market but I have found it interesting enough to devote large amount of my own computational time to map some of what I believe will be important ideas for investors, engineers and businesses in the next 20 years as this revolution stands to shake the foundation of computation as we know it.
Generating nth term variations of complex imaginary planes from pi phi and sqrt-1
Now, adversarial discriminatory instruments and source code repositories are widely available thru nvidia, gpt and many others which utilise high speed ASIC to carry out high performance computational at a great scale. One natural extension of mining is previous research applications into company resource, such as cloud, and beyond transnational trade and intercorporate co-operation at a block chain ledger level (or in other words blockchain securities and cloud). Although I have been engaged in professional interests such as developing blockchain applications for IPFS and Ravencoin and exchange software for the future of transnational securities, these technical, computational and organisational changes of the market sector has been something I predicted long ago, before cloud and blockchain.
One way of looking at the bizarre theory is that computers are engineered to only have one way of looking at things or one output or ‘perspective’, however with quantum computers many more inputs and outputs can be enumerated effectively creating inconceivably complex and recursive depth. We might expect the next generation of gan to have a multi-dimensional logic and intelligence that surpasses current engineering principles utilised presentlyin ray tracing, computational analysis etc completely changing the predictive and creative processes and our relationship with these systems. Certainly for this researches we are interested and fascinated with how human perception and behaviour form and emerge in order to create an engineered analogue of the same
In the early days when divx 3.11 was still new, I remember seeing the possibilities of youtube and netflix, but this feels exponents way bigger in scope. That said the once ceo of google mr schmidt (hope I spelled this correctly) the ability to execute in bulk has become more important in todays modern age than the ability to innovate and invent.
The situation with AI and ML is that it may become in the not so distant future much more important to innovate thru execution of AI and ML processes, which represent in business terms a major change ahead for the world as hybrid machine learning computational systems furnish augmentations to user level behaviour, which organisations that cannot adapt to the increase in tooling, innovation and computational power may perish to. This probably has fascinating social implications but it’s well beyond the scope of this piece. I shall focus on some of these wonderful equations as we try and expand our imagination into 3 dimensional and 4 dimensional generative adversarial relational computation. Certainly we may expect to see an AI and ML black box physicist telling us impossible things about these beautiful relationships that we could not even possibly comprehend or resort to time ofe centuries to discover.
Their stems understandable if not valid reason why there is some measure of concern by the crowd and governance to stem the growth of systems which are beginning to mimic consciousness and intelligent systems in a way that is convincing, which has wide implications for how it will alter human capability and behaviour in the market place. Again it’s really beyond the scope of the piece but we’re interested in food for a computer system that can imagine and we can do this convincingly with generative adversarials because they are not ordinary input and outputs, in fact they re-use outputs to increase the resolution of modeling in a convincing way not dissimilar from human beings. Still as a man who foresee’s power as a measure of universal computational ability in the not so distant future, the social and economic implications are of pretty huge interest if not total fascination to me.
If people understand what this really means, they should get excited! I think! Therefore I Infinite GAN Nourishment!
Moreover the massive expansion in gpu and high performance computation, in particular fixed bus width quantum computers has some dizzying applications for the generative adversarial networks of the future both for use in the financial backtesting and predictive systems. They probably represent a new meta for trading, computation and generative simulation of the future, substantially beyond an order of magnitude ahead, redefining the usual static outputs available in programmatic computation , ray tracing and statistical analysis today, reducing latency of computational to null for inconceivably complex plane solutions, and reshaping the meaning(and limitations) of a computational system we presently know. As – much like human beings, quantum computers will have adequate infinite cycles at a fixed bus width, therefore the computer of tomorrow will look more like a GAN of nearly infinite inputs and reasuable modular outputs in many different factories or ‘faculty’ of ‘cloud idea’ as ‘pools. Then one might expect, perhaps, all those extra cpu cycles, machines will need to ‘learn’ to dream to make good use of the extra cycles and lower latency to gain efficiency of their advantage.
Therefore, In order to plan for such an eventuality the engineer, inventor and ‘musician’ must create the necessary equations to feed such a computationally vast processor, and develop further a working theory of emergent intelligence by mapping human behaviours sufficiently for their recreation etc.
Computational Glyph Searching, a Theoretical Concept introduced in cosmicOS
One approach for nourishment of such an increasingly hypothetical array is the complex imaginary component also known to mathematicians as the square root of -1, whereby a single system or symbolic understanding of related data or informational knowledge can be recursively computed although at fixed bus widths in quantum processors infinite cycles per second, will produce a multitude and magnification of the available computational power and for drawing and navigating intelligence around complex diagrams and performing even more complex discriminations, approaching what we might refer to as the singularity in modern literature, but moreover and most importantly approaching an infinite recursion or divide by zero as the input and ouput cycles on such a hypothetical system as cosmic OS, begin to do more than merely mimic the activities of human inputs and outputs, but evolve with recursive input and outputs which eventually form independent systems which may match or exceed human ability. Naturally such a theory represents great interest to me both as an business minded individual and as a person with a deep interest in distributed computation, generative systems and AI.
CosmicOS has been a labor of love and although ambitious I can see and feel it’s potential. A real engineer wants not just to produce something fantastical and in this case nearly impossible, he wants the research or idea to imbibe meaning and have an impact on others imagination not only to remind him and encourage him that his ideas may have promise or to develop them, or be recognised that his predictions merit, but so that he can see it benefit folks in a beneficial way. A lot of the inventors of the most amazing things don’t receive the credit, and in my estimation probably don’t want it either. I think that’s because most of those folks aren’t just trying to get respect or understanding. They are trying to make the world better, increase human energy and imagination and the only way they can do that is by risking to imagine tomorrow in new impossible and creative ways.
I felt it was time to share more about what I’ve been doing with the possibilities of producing adversarial networks capable of producing large chains of inputs and outputs, and nourishing data of infinite recursive complexity, much like the human brain, since technology has probably caught up enough for people to see the theoretical possibility, and reduced theoretical impossibility of producing all 2^52 combinations for an audio CD for the purpose of universal exploration, researches, ray tracing. Really the goal will be how to sort the data effectively and indeed for sanity and storage purposes recreate nth partition sector of storage programatically using computation rather than a disk- i loosely will term this algorithmic storage – but its a bit too futuristic to encourage others to give it serious thought yet, perhaps.
The applications really are mind boggling and endless for anyone who can understand them. Our biggest problem may be feeding AI and ML Systems, or showing them how to create data to feed them indefinitely would be ideal. Though – if people understood these things better and the things human beings produced and the way the parts that make them up produced them, conscious computer systems would have already been considered long ago as an emergent property of quantum physics and integrated relativistic dimensions of space time.
From mathematical and statistical perspective adversarial applications for deep learning seem substantial
Conclusion – CosmicOS predicates a machine that can observe and reobserve discriminately and reprocess outputs as new inputs to develop ‘pools’ or ‘organs’ of relational intelligence
That kind of predicts a new type of database system collation required to adequately map human memory and creativity! whew! I hope that this provides some insight into the possibilities of GAN, and that it is as always with this industry – still really early in its adoption but it has been making me think a lot about how strange the future will be as computers gain these capabilities.
These developments are not just worthwhile to significantly improve human capability and scientific engineering, but have genuine artistic, scientific and creative applications that will likely one day go far beyond the human imagination. Fortunately today at least people can see these engineering ideas which were predicted hundreds if not thousands of years ago because the technology is finally available to be convincing that people can really see the benefits. Whilst many focus on the drawbacks and danger of machines I feel that it is human beings investments and interests that influence their decisions. These computer systems , theoretical as they may be, I can see will one day be clearly capable of influencing us in ways even I can’t imagine, and I wonder what kind of world that will be. And whether these are issues of human morality and virtue or something deeper as to the complexity of our universe. Whichever is true I shall be equally astonished in my equanimity and enjoyment of these equations and myriad possibilities for the future. So, I suppose an enterprising person then may benefit from acting accordingly on this knowledge in his researches and adventures.
It’s good to keep dreaming and find new ways to express ideas in new projects like cosmic OS. It’s the most fun I’ve had in years.
The ideas are pretty far out for cosmic OS but their ambitiousness is not without some technical meat. The equations work. Is that what matters? yes. It’s probably important that people understand what a GAN is and what try {01} for all parts of a given spacial area represents for a recursive model. It’s… beyond even me, that is why I am so interested in it, but I see the possibilities and it feels good to encourage others and talk more about some of my fun-semi-commercial researches.
Thanks for Reading
Cosmic OS is something I’ve been musing over for decades. Since I first started using distributed compute. I thought it was time to talk a bit more about the idea, and possibly be lambasted for it, but if you don’t take the risk of being wrong you never take the risk of being right. That to me personally has always seemed like too significant a risk of missing. The path of creativity and impossible ambition is the road less voyaged, but one cannot achieve something so ambitious without taking such improbable decisions and calculations with just being ambitious and seeing where it will lead.
I developed a universal translator system that learned languages that don’t exist using the square root of -1. Here is one impossible theoretical diagram for the day. So, I don’t know where a lot of my more unusual ideas come from, but that is another conversation to have probably, the important thing is that the equations matter that prove the discrimination of algorithm is there for discriminated AI/ML output? right? right?
Perhaps.
As an engineer it seems that it well beyond our undertanding of computation presently to really understand what these implications are. Certainly it is a marvelous thing to produce ancient languages from the imaginary complex plain (much like producing audio conversations of caesar) without being there, never needing to leave ones home or rack of computational machinery, with sufficient processing power and storage beyond the number of atoms visible in the universe, it would be a simple matter of known physical engineering to achieve. Despite that it would take several universes to find the conversation, a GAN, and algorithmic finding discovery represent the new brute force generative discriminator adaptor to building new intelligent analysis networks and models far beyond what we have ever conceived. I don’t think I’ll be the guy to do that, but if someone likes the idea there is always a foot note below your project for the credit of the idea.
I’ve been trying out the trial of Wolfram Mathematica as part of my research into cosmic OS and creating a generative universe from the square root of -1 and other interesting things. In this example I’m generating some waves and altering a given component of the equation to provide frame by frame output of the plot. It’s really neat and I am having a lot of fun with it.
I noticed their documentation is pretty useful at documenting each function but you have to do a bit of effort to understand properly how to plot into export directly. The two choices available seem to be a frame by frame fps ‘da’ variable, the other by calling AnimationDuration->15, notice also in this example imageresolution and antialiasing are set, ImageResolution -> 800, Antialiasing -> True, this overides the default animation setting, which for our purposes is too small for a large range of variable sine wave ‘a’.
Stuff like this makes me really happy, and will be a perfect approach in the meantime for some of our renders for Cosmic OS.
