A room labeled "Bing HQ." Developers huddled around a computer, looking confused. The screen shows jumbled text results from Bing Image Create. One developer says, "It's supposed to generate images, not this gibberish!"
Good Reads​
2023-10-04 The Workflow Pattern
found in Programming Digest
2023-09-21 Falsehoods programmers believe about time, in a single list
- There are always 24 hours in a day.
- February is always 28 days long.
- Any 24-hour period will always begin and end in the same day (or week, or month).
- A week always begins and ends in the same month.
- A week (or a month) always begins and ends in the same year.
- The machine that a program runs on will always be in the
GMTtime zone.- Ok, that’s not true. But at least the time zone in which a program has to run will never change.
- Well, surely there will never be a change to the time zone in which a program hast to run in production.
- The system clock will always be set to the correct local time.
- The system clock will always be set to a time that is not wildly different from the correct local time.
2023-08-24 How to sabotage your salary negotiation efforts before you even start
Exactly what to say For questions about comp expectations at the beginning of the process: At this point, I don’t feel equipped to throw out a number because I’d like to find out more about the opportunity first – right now, I simply don’t have the data to be able to say something concrete. If you end up making me an offer, I would be more than happy to iterate on it if needed and figure out something that works. I promise not to accept other offers until I have a chance to discuss them with you.
For questions about comp expectations at the end of the process: It sounds like there’s an offer coming, and I’m really excited about it. I’m not sure exactly what number I’m looking for, but if you’d be able to share what an offer package might look like, then I will gladly iterate on it with you if needed and figure out something that works. I promise not to accept other offers until I have a chance to discuss them with you.
For questions about where else you’re interviewing at the beginning of the process: I’m currently speaking with a few other companies and am at various stages with them. I’ll let you know if I get to the point where I have an exploding offer, and I promise not to accept other offers until I have a chance to discuss them with you.
For questions about where else you’re interviewing at the end of the process: I’m wrapping things up with a few companies and in process with a few more. I promise to keep you in the loop, and I promise not to accept other offers until I have a chance to discuss them with you.
2023-08-15 Book đźš© Software Engineering at Google
We’ve found that expertise and shared communication forums offer great value as an organization scales. As engineers discuss and answer questions in shared forums, knowledge tends to spread. New experts grow. If you have a hundred engineers writing Java, a single friendly and helpful Java expert willing to answer questions will soon produce a hundred engineers writing better Java code. Knowledge is viral, experts are carriers, and there’s a lot to be said for the value of clearing away the common stumbling blocks for your engineers.
Bing!!! A sleek, modern design showcases a vast network of interconnected nodes, symbolizing software intricacy, over a satellite view of Earth. At the center, the bold, white text "Software Engineering at Google" contrasts with a deep blue background, signifying global technological dominance.
Layered over an abstract representation of code, glowing in Google's iconic colors, sits a polished chrome 'G'. Above it, the title "Software Engineering" is written in modern font, with "at Google" just below, emanating the innovative essence of the tech giant.
More Wisdom​
2023-10-04 A list of 100 opinions I hold
- 09 - Log all the things. If you didn’t log it, it didn’t happen.
- 10 - Don’t assume users are dumb. They will find things you never knew was possible.
- 11 - Spend most of the time writing the failure paths, not the happy path.
- 12 - Don’t slap on an index and call it a day. Check your queries, rewrite them and/or create indices accordingly. And hire a DBA.
- 13 - Javascript is an unreadble async mess.
- 14 - Scrum is bad
- 15 - Kanban is better
- 16 - Just because you say you do scrum, doesn’t mean you do scrum.
- 17 - you can be “agile” without daily standups, sprints, retrospectives.
2023-09-27 wisdom/wisdom.md at master · merlinmann/wisdom
Sometimes, an email is just a way to say, “I love you.”
People think about you much less than you either hope or fear.
It’s often easier not to be terrible.
Buy the nicest screwdrivers you can afford.
Every few months, take at least one panorama photo of your kid's room. At least annually, secretly record your kid talking for at least ten minutes. I promise you'll treasure both, and then you will curse yourself for not having done each way more often.
Most well-written characters have something they want—or something they think they want. The more fascinating characters also have something they don’t want you to know. The best ones also have something they’re not pulling off nearly as well as they think.
Related: these are each also true for real people.
2023-09-14 Work Chronicles - Webcomics
View All Comics Archives - Work Chronicles
Good Talks!​
2023-10-03 "Comedy Writing With Small Generative Models" by Jamie Brew - YouTube
2023-10-02 "Making Hard Things Easy" by Julia Evans (Strange Loop 2023) - YouTube
koalaman/shellcheck: ShellCheck, a static analysis tool for shell scripts RFC 9110: HTTP Semantics New tool: Mess with DNS! mess with dns Float Exposed
2023-09-26 World of Warcraft's Network Serialization and Routing - YouTube
I am a Principal Software Engineer, if you are misbehaving, I bring you to my office. JAM -- messaging serialization protocol
SOME FINAL TIPS AND TRICKS FOR BEING AN ODD DEVELOPER:
- Practice using empathy with your users, clients, and team members.
- Consider user security, privacy, and accessibility as core design principles throughout the development process.
- Try to foster a culture of learning and experimentation - remember everyone is at a different point in their learning journey.
- Treat customers, clients, and coworkers how you would want to be treated.
- Measure the impact of your work
Projects​
2023-10-03 orhun/daktilo: Turn your keyboard into a typewriter! 📇
On one side, this is a project with a useless (but funny) goal. On the other side -- this is an awesome sample of cross-platform system app.
daktilo ("typewriter" in Turkish, pronounced "duck-til-oh", derived from the Ancient Greek word δάκτυλος for "finger") is a small command-line program that plays typewriter sounds every time you press a key. It also offers the flexibility to customize keypress sounds to your liking. You can use the built-in sound presets to create an enjoyable typing experience, whether you're crafting emails or up to some prank on your boss.
2023-10-03 google/graph-mining
This project includes some of Google's Graph Mining tools, namely in-memory clustering. Our tools can be used for solving data mining and machine learning problems that either inherently have a graph structure or can be formalized as graph problems.
Web, JavaScript​
2023-10-04 Draggable objects
C and C++​
Typescript​
2023-09-30 One Thing Nobody Explained To You About TypeScript - Redd
Delving deeper into configurations, the article illuminates the necessity of nested configurations for different project layers, advocating for as many TypeScript files as there are layers. This granularity is essential to avoid "unleashing hundreds of ghostly types" and ensuring precise type-checking. As development tools evolve, and while frameworks might abstract complexities, it's emphasized that "TypeScript is still your tool," urging developers to grasp its depths and nuances.
Rust​
2023-10-02 Writing a Debugger From Scratch - DbgRs Part 1 - Attaching to a Process // TimDbg
2023-10-02 Writing a Debugger From Scratch - DbgRs Part 2 - Register State and Stepping // TimDbg 2023-10-02 Writing a Debugger From Scratch - DbgRs Part 3 - Reading Memory // TimDbg 2023-10-02 Writing a Debugger From Scratch - DbgRs Part 4 - Exports and Private Symbols // TimDbg 2023-10-02 Writing a Debugger From Scratch - DbgRs Part 5 - Breakpoints // TimDbg - Tim Misiak
2023-10-02 Programming Digest
Amazing Apache Spark​
The time has come to publish everything. Everything I have collected so far
Apache Spark Performance​
2023-10-01 sql - How to measure the execution time of a query on Spark - Stack Overflow
No, using time package is not the best way to measure execution time of Spark jobs. The most convenient and exact way I know of is to use the Spark History Server. 2023-10-01 ✨ Use the extended Spark history server to debug apps - Azure Synapse Analytics | Microsoft Learn 2023-10-01 Plan to manage costs for Azure Synapse Analytics - Azure Synapse Analytics | Microsoft Learn
2023-09-09 Optimizing Spark jobs for maximum performance
Let’s take a look at these two definitions of the same computation:
val input = sc.parallelize(1 to 10000000, 42).map(x => (x % 42, x))
val definition1 = input.groupByKey().mapValues(_.sum)
val definition2 = input.reduceByKey(_ + _)
| RDD | Average time | Min. time | Max. time |
|---|---|---|---|
| definition1 | 2646.3ms | 1570ms | 8444ms |
| definition2 | 270.7ms | 96ms | 1569ms |
Lineage (definition1):
(42) MapPartitionsRDD[3] at mapValues at <console>:26 []
| ShuffledRDD[2] at groupByKey at <console>:26 []
+-(42) MapPartitionsRDD[1] at map at <console>:24 []
| ParallelCollectionRDD[0] at parallelize at <console>:24 []
Lineage (definition2):
(42) ShuffledRDD[4] at reduceByKey at <console>:26 []
+-(42) MapPartitionsRDD[1] at map at <console>:24 []
| ParallelCollectionRDD[0] at parallelize at <console>:24 []
The second definition is much faster than the first because it handles data more efficiently in the context of our use case by not collecting all the elements needlessly.
Apache Spark Azure​
2023-10-01 Azure Storage Data Lake Gen2 Pricing | Microsoft Azure
Data transfer prices for ADLS When you write data into GRS accounts, that data will be replicated to another Azure region. The Geo-Replication Data Transfer charge is for the bandwidth of replicating that data to another Azure region. This charge also applies when you change the account replication setting from LRS to GRS or RA-GRS. View the Data transfer prices on Blobs pricing page.
➡2023-10-01 Azure Data Transfer Costs: Everything You Need To Know • CloudMonitor
Azure data transfer within the same availability zone is free of charge, while data transfer between two different availability zones now incurs a cost of $0.01 per GB. As mentioned earlier, incoming data traffic or data communicating between Azure services within the same region incur no charges. However, charges start to kick in when data is moved across different Azure regions. These charges depend on the amount of data being transmitted and on the zone from where the traffic is originating. For example, if you transfer data between regions within North America (intra-continental data transfer), you will be charged at a $0.02 per GB rate.
2023-08-28 The Hitchhiker's Guide to the Data Lake | Azure Storage
A comprehensive guide on key considerations involved in building your enterprise data lake
Share this page using https://aka.ms/adls/hitchhikersguide
- The Hitchhiker’s Guide to the Data Lake
- When is ADLS Gen2 the right choice for your data lake?
- Key considerations in designing your data lake
- Terminology
- Organizing and managing data in your data lake
- Optimizing your data lake for better scale and performance
- Recommended reading
Azure Data Lake Storage Gen2 (ADLS Gen2) is a highly scalable and cost-effective data lake solution for big data analytics. As we continue to work with our customers to unlock key insights out of their data using ADLS Gen2, we have identified a few key patterns and considerations that help them effectively utilize ADLS Gen2 in large scale Big Data platform architectures.
Apache Spark Code Snippets​
2023-09-30 SharedSparkContext · holdenk/spark-testing-base Wiki\
Instead of initializing
SparkContextbefore every test case or per class you can easily get yourSparkContextby extendingSharedSparkContext.SharedSparkContextinitializesSparkContextbefore all test cases and stops this context after all test cases. For Spark 2.2 and higher you can also share theSparkContext(andSparkSessionif inDataFrametests) between tests by addingoverride implicit def reuseContextIfPossible: Boolean = trueto your test.
How Apache Spark works​
2023-09-27 Big Data Processing in Apache Spark: Serialization - DZone
This post is organized as follows:
- In Section 1, I briefly review Spark architecture and Spark run modes.
- In Section 2, I review what RDD is and what operations we can do on it.
- In Section 3, I review how Spark creates a physical plan out of a logical plan.
- In Section 4, I demonstrate how a physical plan gets executed and where and how Spark tasks get serialized.
- Finally, in Section 5, I summarize all these as simple rules to avoid the exception.
Apache Spark Videos​
2023-09-24 Operational Tips For Deploying Apache Spark - YouTube
2023-08-28 Spark Basics Partitions - YouTube
2023-04-29 MIT 6.824 Distributed Systems (Spring 2020) - YouTube
- MIT Distributed Systems (https://www.youtube.com/playlist?list=PLrw6a1wE39_tb2fErI4-W...) - This is a series of lectures by Robert Morris (co-founder of YC) on distributed systems and their properties. The lectures pick a specific tool/technology (Google File System, ZooKeeper, Apache Spark, etc.) and then discusses it. I've really enjoyed reading the papers and watching the lectures.
Apache Spark Articles​
2023-09-05 Prudent development of Spark jobs in Scala | by Alec Lebedev | Analytics Vidhya | Medium
Good unit testing coverage
In this article we started exploring working with Spark code in Scala from the software engineering perspective. We created a source code repository in Git and configured a CI/CD pipeline for it in GitLab. We integrated the pipeline to push code coverage metrics to CodeCov.io and implemented unit and integration tests to achieve a high level of coverage. In our unit tests, we experimented with object mocking techniques. In the integration test we generated a sample data set and registered it as a table with SparkSession. We enabled Spark integration with Hive in order to allow the test to write transformed data to a Hive table backed by the local file system. In the next article we will continue this exploration by implementing a data conversion for a practical use case.
2023-08-26 Spark Partitioning & Partition Understanding - Spark By {Examples}
FREE ADS!
2023-08-05 Introduction · The Internals of Apache Spark
2023-08-05 Azure Synapse Spark: Add Scala/Java Libraries | DUSTIN VANNOY
2023-04-29 Debugging a long-running Apache Spark application: A War Story
Security holes​
2023-10-01 Lazarus luring employees with trojanized coding challenges: The case of a Spanish aerospace company
The fake recruiter contacted the victim via LinkedIn Messaging, a feature within the LinkedIn professional social networking platform, and sent two coding challenges required as part of a hiring process, which the victim downloaded and executed on a company device. The first challenge is a very basic project that displays the text “Hello, World!”, the second one prints a Fibonacci sequence – a series of numbers in which each number is the sum of the two preceding ones. ESET Research was able to reconstruct the initial access steps and analyze the toolset used by Lazarus thanks to cooperation with the affected aerospace company.
MS-Files​
2023-10-02 ✨✨ 2008!!✨✨ Mini-Microsoft: Achieving Senior Level 63 at Microsoft
Aspects of an L63 Contributor: some random aspects that come to my mind beyond our CSPs:
- They can own a room: they aren't warming a seat but rather can take charge of a conversation and represent such a deep level of knowledge that they gain respect for what they say and earn a good reputation. Their focus stays on accountable results and this person can bring resolution and closure together.
- Expert: They are sought after to be in meetings, for instance, so that good decisions can be made.
- Results-focused: they are focused on getting great results and don't entwine their ego to particular solutions. They don't get defensive if their ideas are revealed to have flaws but rather delight in being able to move to a better solution.
- Leadership: pro-active leadership that convinces team members of the future direction and even helps to implement it. This is a big difference between those who can complain about the way things should be and those you can actually bring it about.
- Solutions, not problems: following up on the above, they aren't complaining about problems on the team but rather implementing and driving solutions.
- Makes other great: the team benefits and grows from the person's contributions. Answers questions from the team, from support, from customers. Knows what the team delivers backwards and forwards. They are a good mentor.
- Influence when they can, scare when they must: they have fundamental skills in influencing people, but if they need to flip into junk-yard dog mode, they can. They don't give up and walk away but rather fight when they need to fight, escalating only when needed and with lots of justification.
- Makes the boss great: if the team and your boss are succeeding because of you, of course you'll be succeeding too.
- Not doing it for the promotion: if you're out for a promotion, don't do work specifically chose to get the promotion. This is like meeting the Buddha on the road. If you come up with a pretty plan to justify your promotion, you've already lost it. Such plotting is obvious and actually detrimental to your career. If, however, you've determined what it takes to have a successful career in your group at Microsoft and have started what you need to start and stopped what you need to stop, then you're on the right path.
2023-10-02 Interview with an Escalation Engineer at Microsoft - YouTube
We must learn how to debug!
Seattle​
2023-09-30 Unintentional Drug Overdose Data (SUDORS) | Washington State Department of Health
2023-09-30 WA rolls out new data dashboard on fatal overdoses | The Seattle Times
Factorio research​
2023-08-28 R-O-C-K-E-T/Factorio-SAT: Enhancing the Factorio experience with SAT solvers 2023-08-28 teoxoy/factorio-blueprint-editor: A feature-rich Factorio Blueprint Editor 2023-08-28 FactorioBlueprints/factorio-prints: factorioprints.com 2023-08-28 factoriolab/factoriolab: Angular-based calculator for factory games like Factorio and Dyson Sphere Program 2023-08-28 deniszholob/factorio-cheat-sheet: Cheat Sheet for Factorio
2023-08-27 Alternative Friday Factorio Fan Facts
2023-08-27 AlternativeFFFF/Alt-F4: Alternative Factorio Friday Fan Facts, also known as Alt-F4
2023-08-27 efokschaner/terraform-provider-factorio: The Terraform Provider for Factorio
2023-08-27 teoxoy/factorio-blueprint-editor: A feature-rich Factorio Blueprint Editor
2023-08-27 drewtato/factorio-css: A CSS framework to make your things look like Factorio