The central question I started with was whether autonomous vehicles can coordinate with each other at an unsignalized intersection without depending on traffic lights, cloud control, or a central roadside unit. At first this looks like an AI problem, because the vehicle needs perception, planning, and control. But when I looked closer, the hard part was not only deciding what one vehicle should do; it was deciding how multiple vehicles should agree on a shared action under time pressure.

Humans already do a loose version of this. At a signalized intersection we mostly obey the light, but when there is no signal we coordinate through arrival order, road rules, emergency-vehicle priority, hesitation, eye contact, or small gestures. That informal negotiation is basically a consensus problem. For autonomous vehicles, I wanted to explore whether that negotiation can be replaced with a deterministic vehicle-to-vehicle protocol.

Autonomous vehicles like Tesla and Waymo are already strong at many perception and control tasks, but intersections still create difficult edge cases. In those cases, relying only on probabilistic decisions from isolated agents feels weak. A better direction is to let vehicles communicate their intent, agree on a pass order, and then execute that decision consistently.

There are already real examples showing why this matters. During a San Francisco blackout, reports showed Waymo vehicles blocking roads after traffic signals stopped working. I am not using that as a criticism of one company; I am using it as a reminder that autonomy needs coordination when infrastructure becomes unreliable. You can read one of the reports here.

This problem is not limited to vehicles. Any autonomous system where multiple agents share physical space eventually needs mutual decision-making. Warehouse robots, drones, delivery bots, manufacturing robots, and autonomous vehicles can all be locally intelligent, but local planning does not automatically produce safe group behavior. If two agents both decide that their own next move is valid, the system can still deadlock, block a path, or create unsafe motion.

This short clip shows the same coordination failure at a smaller scale. It is not from my vehicle simulation, but it captures the reason this research is interesting beyond V2V. The issue is not only whether one autonomous agent can sense and plan; the harder question is whether a group of autonomous agents can agree on what should happen next when they share the same physical space.

Watch the clip: autonomous robots stuck in a coordination conflict.

Why intersections?

I chose unsignalized intersections because they are a good baseline problem. They are small enough to simulate and reason about, but still hard enough to expose the issues that matter in real deployments: wireless packet loss, changing membership, partial visibility, fairness, priority handling, and real-time decision-making. If a coordination protocol cannot work in this constrained setting, it is unlikely to work in a larger autonomous traffic environment.

There is also a clear safety motivation. The 2023 FARS table below shows 11,843 intersection fatalities in traffic crashes, which makes intersection coordination a concrete safety problem rather than only an academic distributed systems exercise.

Figure 1: 2023 FARS intersection fatality data used to motivate why intersection coordination is a safety-relevant problem.

At a high level, every vehicle has local perception. It can observe its own lane, position, speed, acceleration, and nearby vehicles, but local perception is not always enough. At an intersection, the safety of one vehicle depends on what other vehicles intend to do next, so the decision needs to be coordinated rather than purely local.

This is where the pipeline changes. Traditional autonomous driving is often discussed as PCDA: Perception, Collection, Decision, Action. Prior work in this area, including this IEEE paper, frames the coordination problem as PICA: Perception, Initiative, Consensus, Action. The important addition is the consensus stage, where a vehicle does not simply decide alone; it participates in a protocol that produces a shared decision.

What is the goal?

The goal of this project was to evaluate whether a group of autonomous vehicles approaching an unsignalized intersection can exchange enough state to agree on a safe pass order with low latency and low wireless overhead. The output of the system is a pass queue. Once that queue is committed, vehicles execute the crossing order instead of independently guessing who should move first.

Each vehicle proposal contains the fields needed to reason about the next crossing decision:

• vehicle id
• lane id
• position
• speed
• acceleration or deceleration
• expected turn direction
• priority flag, for example ambulance or police vehicle

This sounds straightforward with four vehicles, but it becomes much harder with 20 or 50 vehicles because the communication layer becomes part of the algorithm. In a data center, consensus protocols can assume relatively stable links, controlled membership, and reliable infrastructure. In vehicle-to-vehicle communication, the network is wireless, shared, lossy, and changing while the protocol is running.

That network constraint is the biggest bottleneck. The system needs decisions on a sub-second timeline, but it cannot depend on data-center style infrastructure. Vehicles communicate over broadcast wireless channels, where messages can collide, delivery can be asymmetric, and two vehicles in the same physical intersection may not hear the same set of neighbors. That makes V2V coordination an edge distributed systems problem, not just an autonomous driving feature.

For this project, I used IEEE 802.11p / WAVE as the wireless model. 802.11p is the Wi-Fi-derived vehicular communication standard behind DSRC-style V2V and V2I communication. It is designed for short-range, low-latency exchange between vehicles and roadside units, usually in the 5.9 GHz ITS band, and it supports infrastructure-less broadcast. That matters for this problem because vehicles at an unsignalized intersection should not need to associate with an access point or wait for a cellular round trip before exchanging safety-critical state.

There are newer V2X directions as well. C-V2X introduced cellular sidelink communication, 5G NR-V2X extends that direction for more advanced connected and automated driving use cases, and IEEE 802.11bd is the next-generation evolution of the 802.11p family. These technologies are important, but the goal of this project was not to benchmark radio standards. I chose 802.11p because OMNeT++ and Veins have mature support for it, it gives a realistic lossy broadcast model, and it is a conservative baseline for testing whether the consensus protocol itself can survive wireless contention, packet loss, and partial visibility. If the coordination layer cannot work over this kind of constrained V2V channel, moving to a newer radio will not automatically solve the distributed systems problem.

Simulation setup

To test this properly, I built my application logic on top of a simulation stack using OMNeT++, Veins, and SUMO. SUMO handles the mobility side: vehicles entering lanes, slowing down, stopping, turning, and crossing the intersection. OMNeT++ and Veins handle the network side, including WAVE / IEEE 802.11p communication, which is closer to realistic vehicular wireless behavior than an ideal in-memory network.

Figure 2: SUMO mobility simulation with an OpenStreetMap-based road network and vehicle movement around the intersection.

The simulation includes:

• IEEE 802.11p broadcast messaging
• channel contention
• packet loss
• wireless latency
• calibrated urban path loss
• Nakagami multipath fading
• building and vehicle obstacle shadowing
• SINR-based packet delivery
• OpenStreetMap-based road layouts

I collected telemetry from each run so that I could evaluate the protocol as a system rather than only as an algorithm. The telemetry included vehicle position, speed, acceleration, lane, turn intent, message latency, packet delivery, decision latency, total wait time, and sent/received message counts. The point was not just to implement a protocol and assume it works; it was to observe where a data-center-style protocol starts breaking in a fast-moving wireless edge environment.

Baseline: all vehicles participate in consensus

The first version was intentionally simple. Every vehicle broadcasts its status, collects the status of other vehicles, and participates in a consensus process. I used Raft as the baseline because it gives a clean structure for leader election, quorum acknowledgement, and committing a decision.

In this setting, the simplified Raft-style flow is:

• elect a leader
• have the leader collect status from participants
• compute the next pass order
• commit the decision after quorum acknowledgement, where quorum means a majority of participants, roughly 51 percent or (n / 2) + 1 votes
• broadcast the committed decision back to the group

Figure 3: Baseline all-vehicle Raft flow where every vehicle participates in status collection, consensus, and quorum acknowledgement.

This was a useful baseline, but it also exposed why directly applying data-center consensus to vehicles is difficult. Raft was designed for a very different environment: relatively stable nodes, reliable links, and membership that does not change every second. Vehicles at an intersection are not data-center nodes, and as the number of vehicles grows, every participant has to send more, receive more, reply more, and wait for more acknowledgements.

The wireless plots were useful while debugging because they made the network behavior visible. Channel utilization showed when vehicles were actively using the shared WAVE channel, and in these runs it stayed close to 50 percent, so the channel was busy but not in an obviously broken state. SINR showed how signal quality changed over time. Together, these plots revealed an important point: the issue was not simply that the channel was always saturated or that SINR was always terrible. Even when the channel did not look completely overloaded, the protocol could still fail because vehicles did not share the same communication neighborhood.

Figure 4: Channel utilization over simulation time, showing when vehicles are actively using the shared WAVE wireless channel.

Figure 5: SINR variation over time, used to understand signal quality and packet delivery behavior in the wireless model.

As density increased, the all-vehicle design became increasingly expensive. More vehicles meant a larger consensus group, more status messages, more replies, and more quorum pressure. The protocol was not only doing more work; it was doing more work on a wireless channel where delivery is already fragile.

Figure 6: Scaling issue in the all-vehicle approach as more vehicles enter the intersection and the consensus group keeps growing.

What went wrong with the vanilla method

With a small number of vehicles, all-vehicle Raft works reasonably well. The problem appears as vehicle density increases. Wait time grows, and the number of received messages grows aggressively because every vehicle is trying to participate in the same coordination process. At that point the protocol is no longer just computing a pass order; it is also creating wireless load that makes the pass order harder to commit.

Figure 7: Wait-time CDFs and message growth for vanilla all-vehicle Raft under increasing vehicle density.

The reply delivery rate shows the issue even more clearly. At 4 vehicles, reply delivery is perfect. At 8 vehicles, it is still high. By 16 vehicles, it is already close to the risk region, and at 20 vehicles the median plotted delivery rate drops to 52 percent, with some minimum values going below 40 percent. That is a dangerous region for a quorum-based protocol because Raft needs a majority of votes to commit. Once delivery is hovering around the 50 percent boundary, and sometimes falling well below it, a small amount of loss or asymmetry can stop the leader from collecting enough acknowledgements on time.

Figure 8: Reply delivery rate drops as fleet size increases, making quorum-based coordination harder in the all-vehicle design.

Adding more participants does not automatically make a wireless consensus decision stronger. If the additional participants increase contention and reduce reply delivery, they make the decision harder to finish. In the all-vehicle design, the system was asking every vehicle to participate in a process that only a smaller set of vehicles actually needed to decide the next crossing order.

The next problem is communication range. The image below shows the basic issue: every vehicle has a limited wireless neighborhood, so broadcast does not mean "everyone in the intersection receives the packet." A vehicle near the center may hear several approaches, while a vehicle farther back in one lane may only hear vehicles on its own side of the intersection. Vehicles on opposite arms may be outside each other's reliable range, and buildings, other vehicles, multipath fading, and road geometry can reduce delivery even when they look physically close.

Figure 9: Wireless communication range creates partial visibility because each vehicle only receives broadcasts from the vehicles inside its effective radio neighborhood.

This means each vehicle builds its own local view of the intersection. One vehicle may know about nine nearby proposals, another may know about six, and another may only know about the vehicles directly ahead or behind it. For all-vehicle Raft, that is a serious problem because the leader is trying to collect acknowledgements from a membership set that may not be equally visible to every participant. Some nodes may never receive the request, some replies may not reach the leader, and different vehicles may believe different sets of vehicles are currently active at the intersection.

So the failure is not only "too many messages." The deeper issue is that the communication graph is local and uneven. Consensus assumes a group that can exchange enough messages to agree, but wireless range turns the intersection into overlapping neighborhoods instead of one fully connected group. This is why forcing every vehicle into one consensus group increases delayed replies, missing acknowledgements, and split views. Wireless range is not a minor networking detail here; it is one of the main constraints shaping the protocol.

Broadcast can hurt, but it can also help

At first, broadcast looked like the problem. If every vehicle broadcasts and every other vehicle replies, the channel gets crowded, and more vehicles create more contention and packet loss. But broadcast also has a useful property: many vehicles overhear the same information, even when that information does not reach every node directly.

That observation changed the direction of the protocol. Instead of treating broadcast only as a source of congestion, I started using it as a way to spread proposals through gossip. A vehicle that misses a direct proposal may still learn it from another vehicle that heard it, which helps repair partial visibility without requiring every vehicle to become a voting participant in consensus.

The new idea was to use gossip for visibility and selective clustering for agreement. In the current design, selected vehicles are mainly front-of-lane or lane-leader vehicles, because they represent the next movement from each incoming lane. Instead of 20 vehicles participating in Raft, the cluster can stay bounded around four participants for a four-way intersection.

Figure 10: Gossip helps spread missing proposals and lets lane leaders build a more complete view without making every vehicle a Raft participant.

If there are not enough lane leaders available, the system can fall back to inviting additional vehicles to participate so that the consensus group still has enough members for fault tolerance. The full fleet still receives the final pass queue, but the expensive decision process is handled by the small group of vehicles that matter most for the immediate crossing decision.

Final protocol

The final protocol has three main parts: perception, consensus, and action. Vehicles first generate and broadcast signed proposals, then lane leaders use the gossiped proposal set to run a bounded Raft decision, and finally the committed pass queue is broadcast and gossiped back to the fleet. The key difference from the baseline is that consensus is scoped to a small group instead of being run across every vehicle.

Figure 11: Final protocol design combining perception, signed proposal broadcast, gossip, lane-leader Raft consensus, and pass-queue execution.

The flow looks like this:

1. Each vehicle reads local perception data from sensors and simulation state.
2. It creates a signed proposal containing lane, position, speed, acceleration, turn intent, and priority.
3. Vehicles broadcast proposals over WAVE.
4. Other vehicles receive proposals and verify signatures.
5. Missing proposals are spread through gossip.
6. Lane leaders are selected for the consensus group.
7. The selected group runs Raft to agree on a pass queue.
8. The pass queue is broadcast back to the full group.
9. Vehicles gossip the decision to nodes that may not have heard the first broadcast.
10. Vehicles execute the pass order.

The protocol also includes several engineering pieces that are important in the wireless setting: message compression to reduce load, public/private key authentication for vehicle identity, application-level CSMA/CA to reduce message collisions, fallback cluster invitation when there are too few lane leaders, and batching to improve fairness while reducing repeated computation. The important point is that every vehicle can share state, but only a bounded set of vehicles participates in the core decision.

How the decision algorithm thinks

The decision algorithm takes proposals and converts them into a pass queue. It collects vehicle proposals, groups vehicles by lane and movement, selects front candidates, accounts for priority vehicles, batches compatible movements, and commits the next pass queue through Raft. This is deliberately simple: the point of the project was not to invent a complicated traffic scheduler first, but to understand whether the coordination protocol could survive realistic wireless behavior.

Batching matters because a strictly one-by-one pass order can be safe but inefficient. If two vehicles do not conflict, the system should be able to schedule them together. Fairness also matters, because a protocol that always favors one lane due to better wireless conditions is not acceptable. The decision logic has to balance priority, arrival order, and lane progress.

Results

The main result is that clustering reduced the size of the expensive consensus group. In all-vehicle mode, the Raft participant count grows with fleet size; with 20 vehicles, all 20 can become part of the decision process. In cluster mode, the Raft participant count stays around four because only front-of-lane vehicles participate.

Figure 12: Cluster mode keeps the Raft participant count bounded while all-vehicle Raft grows with fleet size.

This bounded cluster size is what keeps reply delivery stable. In the plotted runs, cluster mode stayed at 100 percent reply delivery while all-vehicle mode dropped as density increased. The wait-time and message-load results show the same pattern: cluster mode reduced wait time as vehicle density increased and reduced the number of messages each vehicle had to receive.

Figure 13: Cluster mode reduces wait time and received-message load compared with the all-vehicle baseline.

At 20 vehicles, all-vehicle mode had around 1616 received messages per vehicle, while cluster mode had around 535. That difference matters because every extra message competes for the same wireless channel. Reducing received-message load is not only a performance optimization; it directly improves the chance that the coordination protocol can finish within the time budget.

Decision latency also became much more stable. In all-vehicle mode, decision latency increased sharply as the number of vehicles increased, while cluster mode stayed much closer to constant. Throughput improved as well, with cluster mode crossing more vehicles per second than all-vehicle mode, especially at larger fleet sizes. I also tested priority vehicle behavior to make sure that a priority vehicle could benefit from the protocol without completely breaking fairness for normal vehicles.

Figure 14: Cluster mode improves decision latency and throughput while preserving priority-vehicle behavior.

Across these experiments, the overall result was about 2x latency reduction, about 3x throughput improvement, controlled received-message growth, bounded Raft participant count, decision latency around 40 to 50 ms in cluster mode, and 100 percent consensus success in the tested crash-fault tolerant scenarios. The most important point is not just that the numbers improved; it is why they improved. The protocol became aligned with the physical network instead of fighting it.

Takeaways

The biggest lesson from this project is that distributed systems protocols cannot be moved blindly into wireless mobile environments. Raft is a strong protocol in the environment it was designed for, but an unsignalized intersection has different constraints: membership changes quickly, wireless links are unstable, packet delivery is asymmetric, range creates partial visibility, acknowledgements can create broadcast storms, and decisions have real-time deadlines.

The first version failed because it treated every vehicle as an equal participant in one global consensus group. The improved version worked better because it accepted the physical shape of the system. Vehicles are naturally organized by lane, and the next crossing decision only needs the front candidates. The rest of the vehicles need to learn the decision, but they do not need to participate in the core consensus process.

That is why gossip and clustering fit this problem. Gossip repairs missing information across the wider group, while selective clustering keeps the consensus group small enough to make progress under wireless constraints. The same principle can apply to other autonomous systems too: warehouse robots, drones, humanoid robots, and multi-agent AI systems can all run into the same issue where each agent is locally intelligent but the group still needs a coordination protocol.

The broader question is not only whether each agent can plan. It is whether the group can agree while communication is local, unreliable, and time-sensitive. This is what makes the problem interesting to me as a distributed systems problem rather than only an autonomous driving problem.

Limitations

This work currently focuses on crash-fault tolerant behavior. It does not solve Byzantine or malicious vehicle behavior, and it assumes L4+ autonomous vehicles that can understand and execute a committed pass order. It also works best in a homogeneous fleet where vehicles follow the same protocol and trust model, which is easier than a mixed environment where every manufacturer has a different stack, message format, and security model.

There is more work needed on formal safety proofs, larger road layouts, mixed traffic with human drivers, adversarial wireless behavior, and real hardware validation. So I do not see this as the final protocol for autonomous intersections. I see it as a practical experiment showing that if we want autonomous systems to coordinate in the real world, the consensus layer has to be designed with wireless physics in mind.

Conclusion

This project started with a simple question about autonomous vehicles, but it became a project about distributed coordination under physical constraints. The first idea was to run consensus among all vehicles. That was clean, but it did not scale well under realistic wireless behavior because the protocol made the communication problem worse as density increased.

The better approach was to combine gossip with selective clustering. Gossip spreads state, clustering keeps the decision group small, Raft provides agreement inside that bounded group, and the final pass queue is broadcast and gossiped back to the fleet. That combination gave lower latency, higher throughput, and more stable behavior as the number of vehicles increased.

For me, the interesting part is that the same principle may apply beyond vehicles. The core question is how to coordinate many autonomous agents when communication is local, unreliable, and time-sensitive. My current answer is that the protocol should not fight the shape of the environment; it should use it.

The next direction for this research is to make the protocol work under stronger Byzantine settings and connect it with a more complete perception stack. That means moving from "vehicles share honest proposals" toward a system that can reason about faulty or malicious behavior, while still staying fast enough for real autonomous coordination.

If you are interested in this work, want to discuss the protocol, or are working on related problems in autonomous systems and distributed coordination, feel free to email me at mthiruma@ucsc.edu or mati02official@gmail.com.

This blog is designed for students seeking a complete roadmap of chip design in the industry. While I won’t be teaching subjects here, if you have basic knowledge of VLSI and electronics, you can use this blog to gain clarity on the career path you should pursue. I will provide detailed insights into various job roles involved in chip design and fabrication, explaining the responsibilities of engineers at different stages. If you aspire to take on one of these roles, I will also give an in-depth explanation at an engineer's daily work, the tools they use, and the skills required to succeed in that role.

Lets start with some high level perspective of chip design roadmap and then I then i will dive into specifics of each role. There are companies that design the chips(Ex: Nvidia, Micron) and then there are companies that manufacture the chips(Ex: TSMC, Global Foundries) and then there are a few companies that that design and fabricate the chips(Ex: Intel, Samsung). Only a few companies can do both design and fabricate but they are actually good at either fabrication or design not both. Here I will give you in detail explanation of just the design roles I will not be explanation about the roles involved during fabrication of the chips.

Ok lets start with how or who decides how the capabilities of a chip(Ex: CPU, GPU). The chip architect typically decides an ASIC’s capabilities, guided by marketing and system-level requirements, Marketing determines market needs and goals, But these days it is the competition between the companies to be the leader in the chip industry, To be the best and have the fastest/reliable CPU and GPU, The world is moving towards AI adoption in every sector and jobs, and to accommodate this AI boom companies are building massive data centers to handle the AI traffic, Just in the year 2025 all the countries in total have agreed to spend 1.3 trillion dollars, to push AI infrastructure and train better models of LLM. So what that means for the chip industry is the world needs billions and billions of GPU in the next decades and will a continue to grow for a foreseeable future.

I believe we are at the dawn of the new era of semiconductor world that powers AI. Just like how industrial revolution made man power useless, The new AI era is trying to make human thinking useless, Now AI is trying to replace most of the simple/creative jobs. We will see silicon valleys popping up in every major country in the next decade. So if you are looking to build a career in chip design, you have made a good choice. You will have a bright and busy future. Semiconductor is the new oil of the world. Everyone needs semiconductor chips but one who controls the supply will become the superpower of the world.

Ok based on the goal the companies want to achieve, a chip architect decides on the some basic parameters on die area, power, temp, performance

Roadmap

These are the roles and individuals involved in designing advanced semiconductor chips

1. Chip architect

2. RTL design Engineer

3. RTL verification Engineer

4. Physical Design Engineer

5. DFT Engineer

6. Signoff/Tapeout Engineer

This does not mean only 6 people can design a chip from scratch, In the real world it involves hundreds of Engineers all closely working together to design the chip. There are also other roles like Pre-silicon validation engineers, Synthesis Engineer, Physical design verification engineer, Post-silicon validation, Characterization engineers, Firmware/Software integration engineers, Packaging engineers, Test engineers, Analog/Mixed-Signal Design Engineer, Product engineers and many more. Who are also involved in the design process, But I will give a detailed explanation only about the roles I have mentioned in the list, Since they are the main engineers who work on the most complex parts of the chip. I will try to explain the other roles in my second blog.

In this blog, I will provide a comprehensive overview of these six roles, delve into the skills required to pursue each career, and explore what their daily responsibilities might look like. Creating and Fabricate the modern CPU/GPU chips stands as one of the most intricate achievements in human ingenuity. The machinery involved in manufacturing the small node 2nm, 3nm and 5nm are some of the most complex machine ever build by human beings. It costs tens of billions to develop and hundreds of millions of dollars to build some of these machines.

Note: This blog focuses solely on the roles involved in designing a chip from RTL to Tapeout. The processes of fabrication and testing that involve hundreds of additional engineers may be discussed separately in a different blog series.

1. Chip Architect

Role : At the very top of the hierarchy, we have the Architect a pivotal figure in chip design. Think of them as the mastermind or planner of the entire process. They're the ones who set the benchmarks for power, performance, and overall goals for the chip. Essentially, they define the parameters, deciding the acceptable range for factors like area, power consumption, and performance. Guided by marketing and system-level requirements they are the bridge between the management and the engineering teams.

Architect works closely with RTL, Physical and verification teams to the goals are feasible and if it can be met under the timeline. They discuss on the trade offs in terms of power, performance, area, and cost making adjustments to the design as needed. They devise a blueprint of all the blocks and determine how these components should be seamlessly integrated. The Architect is involved and consulted throughout various stages of the chip design process. They also Analyze safety mechanism and hardware software STL (software test libraries) tradeoffs and functional safety architecture solutions, methodology, and certification

Day in the life of Architect :

• Architects spend a significant portion of their time in meetings, discussing solution designs with engineering teams, feasibility with product managers and discuss project updates

• Creating TRM : Technical Reference Manual, Documentation and reference models for verification team. Without these it would be difficult to coordinate between different teams.

• Modify the design document and specifications and Working out the tradeoff's for different implementation decisions or iterations.

• Conduct meeting between different teams of RTL, Physical Design and Verification teams to debug the any issue that may arise.

Skills Needed :

• Technical Skills : Deep understanding of Computer Architecture, Architecture Modeling, Trade-off Analysis, Memory and Interfaces Implementation, ASIC/FPGA/SOC design and implementation, RISC/ARM architecture, Design flow, Digital and Analog Circuit Design

• Soft Skills : Project Management, Communication Skills, Teamwork

• Programming Languages : C , C++, Verilog, VHDL

How to become a Chip Architect : Most roles in this field demand at least 6+ years of industry experience and a Master's or Ph.D. in Computer Engineering, Electrical Engineering, or Computer Science. Becoming a Chip Architect right after graduation is exceptionally challenging. As this position sits at the top of the hierarchy, it requires substantial experience, a strong foundation in teamwork, and a track record of success to thrive in the role. For recent college graduates, aiming to become a Chip Architect within five years can be a realistic and ambitious goal.

You should focus on projects/thesis on actual architecture, try to publish in architecture conference like ISCA, MICRO, HPCA, ASPLOS. Read papers from those conferences to get an idea of the current state of research/industry. The best way to start is to apply for positions in performance modeling for a architectural team and start building your way up.

Since Power and Performance are at the core of design, Best way is to start there. You should do a few project about Power/Performance to get a clear idea on how your future work might be.

Note: Skills and requirement will change according to the company, But I have mentioned everything I have researched.

Pic Courtesy: Chip-Architect

2. RTL Design Engineer

Role : An RTL (Register Transfer Level) Design Engineer is a specialist responsible for translating the chip architect's high-level blueprint into a synthesizable RTL description using hardware description languages (HDLs). After the chip architect finalizes the high-level design, the RTL engineer breaks it down into manageable modules, writes and optimizes the code, and ensures the design meets performance, power, and timing requirements. They also collaborate with verification teams to develop testbenches and build simulations for validating the design.

In large projects, there may be anywhere from dozens to hundreds of RTL designers working simultaneously on different sections of the chip. Their role involves optimizing and preparing the RTL for synthesis, ensuring it can be converted into a gate-level design that adheres to timing requirements and resource constraints. This phase represents the initial stage where the chip begins to take form. Additionally, RTL designers have to work closely with verification and physical design teams to achieve timing closure and meet all design specifications. That being said, in smaller companies, a designer often takes on dual responsibilities, handling both design and verification tasks.

Day in the life of RTL Design Engineer :

• RTL designers spend most of their time writing and debugging RTL code for specific modules and running simulations to verify that design.

• They would Participate in daily meetings to discuss progress, issues, and coordinate with other teams, to ensure a seamless progression of the design through various stages, focusing on discussions around timing, power, and area optimizations.

• Analyzing and resolving errors or warnings reported by their tools, while consulting with colleagues to determine whether specific violations can be waived. RTL coding primarily involves connecting and integrating signals between third-party IPs and certain in-house IPs, typically without direct involvement in high-priority internal modules.

• Documenting every design decisions, code, and verification processes.

Skills Needed :

• Technical Skills : EDA Tools (Synopsys, Cadence, Mentor Graphics), Design fundamentals, FPGA/ASIC design flow of simulation and synthesis, TCL scripting, Computer architecture, Static timing Analysis.

• Programming Languages : Verilog, System Verilog, VHDL.

How to become a RTL Design Engineer : Most roles in this field demand a BS or Master's in Computer Engineering, Electrical Engineering with focus on Digital design and VLSI. With the necessary skills, it is entirely feasible to step into the role of an RTL Design Engineer immediately after graduating.

Students should gain hands on projects, internships, or research labs where you can write and simulate HDL code. Should have a decent exposure to industry EDA tools for Design and Verification. You should work on a few projects related to RTL design to gain a better understanding of whether this is the field you want to pursue.

Participate in competitions/workshops online/offline related to RTL design. With the right skills and projects, becoming an RTL Design Engineer is achievable. However, since this is a highly competitive industry, you need to excel to secure your place. Build a passion for RTL code.

Note: Skills and requirement will change according to the company, But I have mentioned everything I have researched.

3. RTL Verification Engineer

Role : An RTL Verification Design Engineer is responsible for ensuring that the Register Transfer Level (RTL) design of a chip functions as intended according to the architect’s blueprint. Their primary role is to verify that the RTL design meets specifications and operates correctly before proceeding to subsequent stages, such as synthesis or physical design. They create simulation environments and test benches to ensure a reliable design before advancing further in the chip design pipeline. They act as the bridge between the RTL Design Engineer and the Physical Design Engineer by analyzing simulation results, debugging issues, and verify fixes and validating that the RTL code functions correctly according to the design specifications, before it undergoes synthesis.

In large companies, the ratio of RTL Design Engineers to RTL Design Verification Engineers typically ranges from 1:2 to 1:3. This means there are often two to three verification engineers for every design engineer. The larger number of verification engineers highlights the effort needed to thoroughly test and validate complex designs to meet performance standards. Usually there will be a synthesis engineer who optimizes the RTL design into a gate-level netlist representation suitable for further design changes. Verification engineer makes sure the chip will perform reliably once fabricated, reducing the risk of costly post-silicon fixes.

Day in the life of RTL Design Verification Engineer :

• RTL verification designers spend most of their time writing or updating testbenches in SystemVerilog or similar languages While using EDA tools to run simulations and verify the design under various scenarios.

• They review requirements to create a verification plan with an emphasis on test cases linked to specific requirements, working on developing components like predictors or sequences for a UVM testbench. also running random regressions to analyze the results, and report any identified bugs to the designer. Once the issues are resolved, rerun the regression and proceed to verify other features.

• They will participate in meetings to discuss these finding with their team and possible solutions.

• Documenting every design decisions, testbench, and verification processes.

Skills Needed :

• Technical Skills : Design fundamentals, Computer architecture, Verification methodologies, Scripting skills, EDA Tools (Synopsys VCS, Mentor Graphics, Cadence Xcelium), Python/Perl scripting, UVM, FPGA/ASIC design flow of simulation and synthesis.

• Programming Languages : Verilog, SystemVerilog, Python, Perl

How to become a RTL Design verification Engineer : Most roles in this field demand a BS or Master's in Computer Engineering, Electrical Engineering with focus on Digital design and VLSI. With the necessary skills, it is entirely feasible to step into the role of an RTL Design Verification Engineer immediately after graduating.

Students should engage in hands-on projects and internships focused on writing and simulating Verilog/SystemVerilog code, as well as applying verification methodologies. Having exposure to Industry tools is a plus, Should have a good understanding of ASIC/FPGA design flow. Also should develop a passion for writing testbench and automation scripts.

Always participate in competitions/workshops online/offline related to Design verification, It is always a good chance for you to put use of the skills you learnt to the test, If you have all these skills you have a chance in becoming a RTL design verification Engineer. and build a passion for RTL code.

Note: Skills and requirement will change according to the company, But I have mentioned everything I have researched.

4. Physical Design Engineer

Role : A Physical Design Engineer is responsible for translating the digital design into a manufacturable physical layout while ensuring various constraints are met. Their primary role is to implement Partitioning, Floor planning, Placement, Routing, Clock tree synthesis, Static timing analysis of the various blocks of the chip to ensure that the design meets timing, power, and signal integrity requirements. They work to close timing paths, reduce power consumption, and optimize area usage through iterative design and simulations. Their role begins after the chip architect and RTL design engineers have developed the RTL code and synthesized it into gate level netlist by Synthesis engineer.

They play a critical role in preparing the chip for tapeout, handling design-rule checks (DRC) and layout-versus-schematic (LVS) checks to ensure the design is ready for fabrication. Physical design stage is indeed one of the most time-consuming and critical parts of semiconductor chip design as even minor adjustments can significantly impact performance or yield. Despite advanced EDA tools, much of their work involves iterative manual fine-tuning. Even minor errors can lead to costly production issues. The complexity of physical design is often underestimated and most companies employ Physical Design verification team to check for error and run simulation for various conditions, it requires both a broad strategic view and meticulous attention to detail, making it one of the most challenging stages in chip design.

Day in the life of Physical Design Engineer :

• Physical design Engineers spend most of their time working on floor planning and placement using EDA tools, perform timing analysis and routing to ensure optimal performance.

• They would Participate in daily meetings to discuss progress, issues, and coordinate with other teams, focusing on discussions around timing, power, and area optimizations.

• Most of Physical design Engineer time would be spend on running simulation and working around bugs in Innovus or ICC and writing TCL code to automatic something to fix a problem that shouldn't happen in the first place.

The months leading up to the Tapeout are always hectic, as last-minute issues from other teams may require them to work overtime to resolve the bugs and run the simulation.

Skills Needed :

• Technical Skills : EDA Tools( Genus, Innovus, Tempus, Voltus), Physical design flow, Digital design, ASIC/SOC design and implementation, Computer Architecture, Analog/Digital circuit design, PPA tradeoff, Static timing Analysis.

• Programming Languages : Verilog, SystemVerilog, Python, Perl

How to become a Physical Design Engineer : Most roles in this field demand a Master's or Ph.D in Computer Engineering, Electrical Engineering with focus on Digital design and Computer Architecture. With the necessary skills, it is entirely feasible to step into the role of an Physical design Engineer immediately after graduating but it is highly competitive job that has fewer number of openings So you have to be the best in this field to increase your chances.

Students should gain hands on projects, internships, or research labs where you can write and simulate RTL to GDSII. Learning every step of floorplanning, Pnr, STA. Also should have a decent exposure to industry EDA tools and You should work on a few projects related to Physical design to gain a better understanding of whether this is the field you want to pursue.

Always participate in competitions/workshops online/offline related to Physical design, In my next blog I will share more details about the projects/workshops needed for different roles, You can refer those projects to make even better projects using the skills you develop, It is always a good chance for you to put use of the skills you learnt to the test, If you have all these skills you have a chance in becoming a Physical Design Engineer.

Note: Skills and requirement will change according to the company, But I have mentioned everything I have researched.

Pic Courtesy : Wikipedia

5. DFT Engineer

Role : A DFT (Design-for-Test) Engineer is responsible for designs and implementation of test structures into semiconductor circuits. While Physical Design Engineers translate the logical circuit design into a physical layout ensuring the chip meets area, timing, and power constraints, DFT Engineers then incorporate testability features into the design. Their work ensures that once the chip is fabricated, it can be properly tested with automated test equipment (ATE), They are responsible for integrating scan chains, built-in self-test (BIST) structures, and other test methodologies into the chip design. DFT engineers are required before and after fabricating the chip.

Building on the test structures designed by the DFT team, there will be other teams like pre silicon validation team who tests these feature before the tapeout. DFT engineer commonly use BIST/MBIST design methodology to incorporate testing mechanisms directly into the chip so overall test time is reduced. A BIST contains a Pattern Generator that creates test patterns, using pseudo-random algorithms or predefined sequences. The Response Analyzer compresses circuit outputs into a signature to validate functionality. The Control Logic oversees the BIST process, managing test sequences and transitions between test and normal modes. DFT engineer may use TCL script for handling data, So yeah every stage of semiconductor design is complex.

Day in the life of DFT Engineer :

• DFT Engineers spend most of their time working writing and reviewing scripts for scan insertion and BIST for simulation and verification to ensure that test patterns can effectively cover the design and detect potential faults.

• They would Participate in daily meetings with verification and test teams to discuss issues and adapting to the constantly evolving design driven by errors and various other factors.

• They develop automation scripts to improve testing time and debug the issue found during simulation and working those changes with other teams. Spend hours writing scripts and debugging the errors.

• Documenting every DFT methodologies, test coverage metrics and verification processes.

Skills Needed :

• Technical Skills : Digital circuit design, DFT methodologies(BIST, Fault simulation, scan insertion), EDA Tools(Mentor Graphics) , Writing scripts and debugging, DFT Architecture Selection, Computer Architecture

• Programming Languages : Verilog, Python, Perl, Tcl,

How to become a DFT Engineer : Most roles in this field demand a Master's or BS in Computer Engineering, Electrical Engineering with focus on Digital design and semiconductor devices. With the necessary skills, it is entirely feasible to step into the role of an DFT Engineer immediately after graduating.

Students should engage in hands-on projects and internships focused on DFT concepts and debugging, as well as applying DFT methodologies. This is a role where you will write a lot of automation scripts and debug the issue, If you are good at this then you should consider a different role, Should have a good understanding of different DFT methodologies. Also develop a passion for scripting and debugging in order to master this role

Always participate in competitions/workshops online/offline related to DFT, It is always a good chance for you to put use of the skills you learnt to the test, If you have all these skills you have a chance in becoming a DFT Engineer. It's always beneficial to network with people who share a similar background or expertise.

Note: Skills and requirement will change according to the company, But I have mentioned everything I have researched.

6. Signoff / Tapeout Engineer

Role : A Signoff/Tapeout Engineer Engineer is responsible for verifying and finalizing the physical layout of the chip before design is sent for fabrication or as I like to call “ The last line of defense to avoid a costly error during production”. while DFT engineer integrates test structures and after pre-silicon validation engineers have verified the design's functionality. Signoff / Tapeout Engineer work is a crucial final check, before it is sent to the Fab. So this involves ensuring that the layout adheres to design rules, meets timing and power specifications, and is manufacturable. They perform detailed checks and validations (often called “signoff”) before the design is sent for fabrication.

While Signoff and Tapeout engineer role may overlap and often considered as one position in smaller companies but generally Signoff Engineer focuses on DRC/LVS, Antenna checks and meet all manufacturing requirements, Tapeout Engineer focuses on packaging the design while verifying all the design documents and generating signoff reports before sending it to the foundry, They may also act as bridge between the foundry and the engineering team.

Day in the life of Signoff / Tapeout Engineer :

• Signoff / Tapeout Engineers spend most of their time reviewing and updating layout files based on feedback from DRC/LVS reports, and preparing for meetings with the design and verification teams.

• They would Collaborate with colleagues to fine-tune the layout, attending design reviews, and documenting changes as they prepare the signoff reports.

• Scheduling a meeting with the foundry to review the design, address any potential errors, and discuss corrective measures and ensure there are no major production level errors.

• Documenting all changes, metrics and verification processes.

Skills Needed :

• Technical Skills : Digital circuit design, Verifications techniques, Physical design flow, Computer architecture, ASIC/SOC design and implementation, EDA Tools

• Programming Languages : Tcl, Perl, Python

How to become a Signoff / Tapeout Engineer : Most roles in this field demand a Master's or BS in Computer Engineering, Electrical Engineering with focus on Digital design and semiconductor devices. With the necessary skills, it is entirely feasible to step into the role of an DFT Engineer immediately after graduating.

Students should engage in hands-on projects and internships focused on Tapeout concepts and debugging. This is a role where you a lot of attention to detail, clear communication skills, If you are good at this then you should consider a different role also Should have a good understanding of layout design, simulation, and physical verification. Also develop a passion for scripting and debugging in order to master this role.

Always participate in competitions/workshops online/offline related to Design flow, It is always a good chance for you to put use of the skills you learnt to the test, If you have all these skills you have a chance in becoming a Signoff / Tapeout Engineer. It's always beneficial to network with people who share a similar background or expertise.

Note: Skills and requirement will change according to the company, But I have mentioned everything I have researched.

Pic courtesy : Tiny Tapeout

Additional Information : I can’t write everything I know or have researched about a role in just a few paragraphs, If explained fully it would take nearly 10 pages for each one! If you’re interested in these blogs or would like to discuss a specific position, feel free to contact me for a conversation.

There are many roles involved in the designing of these chips that I haven’t mentioned here. Explaining every role from RTL to Tapeout would make this blog around 150 pages long, which is why I plan to write a series of blogs exploring each role in detail.

Please note that the skills and the day-to-day responsibilities of an engineer I’ve mentioned are general and may vary depending on the company. I’ll leave my contact information here. If you’re interested in discussing a role or need more details about a position you couldn’t find online, feel free to reach out I’ll do my best to help. You can find similar information in Chat GPT or other LLM but the information found in this blog is from professionals who work in the industry.

If you find any mistake in this blog, Kindly send me a mail so I can verify the error and make changes accordingly.

Gmail : gbukkapa@ucsc.edu

LinkedIn : www.linkedin.com/in/gowrav-raj

Reference

• Wikipedia: Utilized for broad, crowd-sourced information on semiconductor design roles, providing foundational definitions and context.

• Careers Page: Consulted to obtain detailed role-specific expectations and job descriptions based on established industry standards.

• Industry Professionals: Engaged with experienced professionals to gather real-world insights into day-to-day responsibilities.

• Personal Experience: Incorporated practical, hands-on knowledge and industry-relevant details from my own exposure, reflecting my commitment to a career in semiconductor design.

Convolutional neural networks (CNN/ConvNet) are a kind of deep neural network used most frequently to interpret visual data in deep learning. Normally, matrix multiplications come to mind when we think of a neural network, but that is not the case with ConvNet. It makes use of a unique method called convolution. Convolution is a mathematical procedure that takes two functions and creates a third function that explains how the form of one is changed by the other in mathematics.

Flow of CNN

Convolutional Neural Network or CNN is a type of artificial neural network, which is widely used for image/object recognition and classification. Deep Learning thus recognizes objects in an image by using a CNN.

Layers

Convolution

CNN tests these traits everywhere and in every conceivable place because it is uncertain where they will match in a fresh image. We turn it into a filter by computing the match to a feature over the entire picture. Convolutional Neural Networks get their name from the convolutional math we utilise to do this.

Convolution's arithmetic is simple enough for a sixth-grader to understand. Simply multiply each pixel in the feature by the value of the corresponding pixel in the picture to determine how well a feature matches a region of the image. Divide the total number of pixels in the feature by the sum of the replies. When both pixels have a value of 1 or white, 1 1 equals 1. If they're both black, (-1) (-1) = 1. In either case, a 1 is produced for each matched pixel. Any discrepancy is also a -1. When all the pixels in a feature match, their sum is 1, and their number is divided by the total number of pixels. Similar to this, the response is a -1 if none of the pixels in a feature match the picture patch.

We continue this procedure in order to finish our convolution, aligning the feature with each potential picture patch. On the basis of the location of each patch in the picture, we may take the result of each convolution and create a new two-dimensional array from it. The image we started with has been filtered to create this map of matches. It shows the location of the feature inside the picture on a map. Strong matches are shown by values close to 1, strong matches for the photographic negative of our feature are indicated by values close to -1, and no matches of any kind are indicated by values close to zero.

The complete convolution procedure must then be repeated for each of the additional characteristics. A collection of filtered photos, one for each of our filters, is the outcome. Conveying this entire set of convolution processes as a single processing step is practical. This is referred recognised as a convolution layer in CNNs, indicating that further layers will be added to it shortly.

It's understandable how CNNs gained a reputation for being computation hogs. The number of additions, multiplications, and divisions can pile up quickly even though we can draw our CNN on the back of a napkin. They scale linearly with the number of pixels in the picture, the number of pixels in each feature, and the total number of features, to use mathematical terminology. With so many contributing components, it's simple to multiply this issue by millions without breaking a sweat. It should come as no surprise that chip makers are now producing specialist processors in an effort to keep up with CNN's needs.

Pooling Layer

The size of the feature maps is reduced by pooling layers. As a result, it lessens the quantity of network computation and the number of parameters that must be learned. The feature map created by a convolution layer's feature pooling layer summarises the features that are present in a certain area. Therefore, instead of precisely positioned features produced by the convolution layer, additional actions are conducted on summarised features. As a result, the model is more resistant to changes in the features' positions in the input picture.

Flattening Layer

A flatter layer is a pleasantly straightforward phase in the construction of a convolutional neural network. It entails converting the pooled feature map produced during the pooling process into a one-dimensional vector. Here is an illustration of how this procedure appears:

Fully connected layer

The entire connection stage entails chaining an artificial neural network onto our pre-existing convolutional neural network, as you can probably surmise from the previous part. The artificial neural network's hidden layer is replaced with a particular kind of hidden layer known as a fully connected layer in this stage, which is why it is known as the full connection step. The neurons in a layer that is totally connected are all linked together, as the name suggests. Here is an illustration of an artificial neural network's completely linked layer in visual form:

In a convolutional neural network, the fully connected layer's job is to identify certain characteristics in an image. A specific feature that could be present in a picture is associated with each neuron in the fully connected layer. The likelihood that the feature is present in the picture is represented by the value that the neuron transmits to the following layer. Convolutional neural network and artificial neural network both come to an end at the same time. To put it another way, CNN's artificial neural network predicts what is there in the image that it is attempting to recognise!

See you soon

Turn on your notifications and stay in touch with us to learn about new ideas. Share this blog with your friends and family... I will see you in the next blog. Thank you all.

Previous blog, We learned about what reinforcement is and the types of approaches it can take. Here, we are going to see a very interesting concept called "NLP".

What is NLP

The field of computer science known as "natural language processing" (NLP) is more particularly the field of "artificial intelligence" (AI) that is focused on enabling computers to comprehend spoken and written language like that of humans.

Computational linguistics, or the rule-based modelling of human language, is combined with statistical, machine learning, and deep learning models to form NLP. With the use of these technologies, computers can now interpret human language in the form of text or audio data and fully "understand" what is being said or written, including the speaker's or writer's intentions and mood.

Computer programmes that translate text between languages, reply to spoken commands, and quickly summarise vast amounts of text even in real-time are all powered by NLP. You've probably used NLP through voice-activated GPS devices, digital assistants, speech-to-text dictation programmes, customer service chatbots, and other consumer conveniences. However, the use of NLP in corporate solutions is expanding as a means of streamlining company operations, boosting worker productivity, and streamlining mission-critical business procedures.

Techniques:

Segmentation

In order to identify the lengthier processing units made up of one or more words, sentences must first be segmented. To complete this test, you must recognise the boundaries between sentences and words. Due to the punctuation marks that appear at the ends of sentences in the majority of written languages, sentence segmentation is also known as sentence boundary detection, sentence boundary disambiguation, or sentence boundary identification. Choosing how to segment a text into sentences for further processing is the task that is described by all of these words.

Tokenisation

By identifying the word boundaries the locations where one-word finishes and another begins and tokenizing the text, the sequence of letters in the text is broken up. These words are sometimes referred to as tokens for computational linguistics reasons. Word segmentation, which is sometimes used synonymously with tokenization in written languages where word boundaries are not explicitly defined by the writing system, is another name for tokenization.

Stop word

The terms "stop words," "stop word list," and even "stop list" are frequently used while dealing with text mining software. In general, stop words are a group of frequently used terms in all languages, not only English.

Determiners: Determiners tend to mark nouns in situations when a determiner ordinarily is followed by a noun. Examples: the, a, an, another

Coordinating conjunctions: Coordinating conjunctions join words, phrases, and clauses. Examples include: for, an, nor, but, or, yet, and so.

Prepositions: Prepositions convey geographical or temporal relationships. Some examples are in, beneath, toward, and before.

Stemming

Stemming is a method for eliminating affixes from words to reveal their basic structure. It is analogous to trimming a tree's branches down to the trunk. For instance, the word eat is the root of the verbs eating, eats, and eaten.

Stemming is used by search engines to index words. Because of this, a search engine can only record the stems of a word rather than all of its variations. Stemming does this by reducing the size of the index and improving retrieval precision.

Lemmatization

Similar to stemming is the lemmatization approach. The final product of lemmatization is referred to as a "lemma," which is a root word as opposed to a root stem, the final product of stemming. We will obtain a valid term with the same meaning after lemmatization.

The WordNetLemmatizer class from NLTK is a simple wrapper for the WordNet corpus. To locate a lemma, this class employs the WordNet CorpusReader class's Morphy() method.

Part of Speech (PoS) Tagging

A phrase is transformed into forms using this method, which results in a list of words and a list of tuples, each of which has a form (word, tag). Whether a word is a noun, adjective, verb, etc., is indicated by the part-of-speech tag in the instance.

Named Entity Recognition

The term "Named Entity" was initially suggested at the Message Understanding Conference (MUC-6) to recognise names of businesses, individuals, and places in the text, as well as phrases for money, time, and percentages. Since then, NER and Information Extraction (IE) algorithms using text-based data for numerous scientific events have attracted growing interest.

Applications:

Speech recognition-The process of accurately translating voice data into text is known as speech recognition, commonly referred to as speech-to-text. Any programme that responds to voice commands or inquiries must use speech recognition. The way individuals speak quickly, slurring words together, with varied emphasis and intonation, in various dialects, and frequently using improper grammar makes speech recognition particularly difficult.

Part of speech tagging-Grammatical tagging, also known as part of speech tagging, is the technique of identifying the part of speech of a certain word or passage of text based on its use and context. I can construct a paper plane, and make it classified as a part of speech as a noun in 'What makes of automobile do you own?'

Word sense disambiguation-Choosing a word's meaning from among its potential meanings by utilising the semantic analysis to determine which word makes the most sense in the given circumstance is known as word sense disambiguation. By way of example, word sense disambiguation helps to distinguish between the meanings of the verbs "make," "make the grade," (accomplish), and "make a bet" (place).

Sentiment analysis-Sentiment analysis looks for intangible elements in the text, such as attitudes, feelings, sarcasm, bewilderment, and mistrust.

Natural language generation-Natural language generation is the process of converting structured data into human language; it is frequently referred to as the reverse of voice recognition or speech-to-text.

See you soon

Turn on your notifications and stay in touch with us to learn about new ideas. Share this blog with your friends and family... I will see you in the next blog. Thank you all.

What is Python?

Python is an interpreted, object-oriented, and high-level programming language. It was created by Guido van Rossum in 1991. Python is mainly used to boost the speed of software development by providing an easy-to-learn and readable interface.

What makes Python so popular?

• It is a free and open platform that anyone can use
• Python is user-friendly due to its simple syntax
• Large library and assistance
• It can be integrated with other languages

Applications of Python

Python can be used to develop various kinds of applications such as Web applications, Gaming applications, Machine Learning and Artificial Intelligence, Data science and data visualization, Web scraping applications, etc., All these applications can be built at a rapid speed by using Python frameworks.

What is a Framework?

A framework is an arrangement that gives some basic structure around which you add your code to have a greater functionality of that system. As frameworks are free and open source it allows you to create an application in a standard way.

Below are some of the frameworks used for various applications.

• Web applications - Django, CherryPy, TurboGears, Flask, Web2Py, Bottle
• Gaming applications - Pygame, Pyglet, PyKyra, Kivy, Panda3D
• Machine Learning - Numpy, Scipy, Scikit-learn, TensorFlow, Keras, PyTorch, Pandas
• GUI applications - PyQT, Python Tkinter, PyGTK, PySide2

Before stepping into the learning process, let’s see how to install Python on your machine.

Installation of Python in Windows

Step 1: Go to the website python.org, click on the Downloads tab and click on windows.

Step 2: Select the version of python you want to install based on your system configuration.

Then download the executable (.exe) file.

Step 3: Once downloaded, run the executable file to install python. If you want to customize the location select Customize installation otherwise click on Install Now. Make sure that you select the checkbox at the bottom to add python to PATH.

Step 4: Once the installation process is completed the below popup will appear.

Now, Python is successfully installed.

Step 5: To ensure that Python is installed in your system, Follow the below steps

• Open command prompt
• Type 'python' and hit enter
• This command will give the version of Python installed

If you can see the Python version you installed, Python has been successfully installed and you are ready to use it.

Writing your first python program

To write your first program in Python, Go to start and search for Python. You can see Python IDLE. As you open it, a Python shell will appear.

Then do the following steps

• Go to File>New file
• Type the code - print(“Hello World!”)
• Save the file by adding the extension as .py
• To run the code go to Run>Run module or directly press F5 key
• The output will be displayed in the Python shell

Now we have successfully printed “Hello World!” using Python.

A chrome extension is a program that is installed in the Chrome browser that enhances the functionality of the browser. You can build one easily using web technologies like HTML, CSS, and JavaScript.

Creating a chrome extension is similar to creating a web application, but it requires a manifest.json file which we will discuss in the last section of this post.

What Will our Chrome Extension Look Like?

As you can see, the above chrome extension will create QR code for website where you clicked the extension. We will be looking into how to create this extension in this blog post.

Here, we will be using the https://api.qrserver.com/v1/create-qr-code/?size=150x150&data= API in the order to create QR code.

The complete source code of this project can be found on GitHub.

How to Create a Chrome Extension

First of all, we need to create an empty folder where we will add our HTML, CSS, and JavaScript files.

Inside the folder, let’s create an index.html file with this HTML boilerplate code:

<html lang="en">
<head>


</head>
<body>
    <img src="example" id="qrcode">
  <script src="Content.js"></script>
</body>
</html>

In the above code it will display QR code of 150px X 150px. In the image tag QR code will be displayed.

Now, since the table has been displayed, we need to work on writing JavaScript in order to fetch data from the API.

Let's create a script.js file and add the following code:

chrome.tabs.getSelected(null, function(tab) {
    document.getElementById("qrcode").src = "https://api.qrserver.com/v1/create-qr-code/?size=150x150&data="+tab.url;
});

Now, let’s break down the above code:

• Here we are using the API to create a QR code
• Then it will display it in the Image tag which as 'qrcode' as ID

If we check the browser, we will be able to see the following result.

Manifest.json File

As we discussed earlier, building a Chrome extension is similar to building any web application. The only difference is that the Chrome extension requires a manifest.json file where we keep all the configurations.

The manifest.json file contains all the necessary information required to build the Chrome extension. It is the first file the extension checks and everything is loaded from this single file.

Now, let's create a manifest.json file in the root folder and add the following code:

{
    "name": "VK QR",
    "version": "1.0",
    "description": "demo",
    "manifest_version":2,
    "browser_action":{
        "default_popup":"demo.html",
        "default_icon":"logo.png"
    },
    "background": {
        "service_worker": "Content.js"
      },
    "action":{}
    ,
    "permissions": [
        "tabs",
        "activeTab",
        "scripting",
        "<all_urls>"
    ]
}

Our manifest.json file contains the value of name, version, description, manifest_version (2 in this case, which is the latest manifest version), author, and action fields. In the action field, there's the value for default_popup which contains the path to the HTML file which is index.html in this example.

You can have a look here to see all configurations of a manifest.json file.

Now, since we've also added the manifest.json file we are ready to add this project as an extension in our Chrome browser.

For that, we need to go to Select More Tools and then choose Extensions from the browser menu as shown in the picture below:

After choosing Extensions, it redirects to the extensions page in Chrome. Make sure to enable the Developer mode here.

Once that's done, you need to click the Load unpacked button which will allow us to load our project in the Chrome extension store.

Now, the extension is available in our Chrome extension store. You can also pin the extension in the browser as shown in the gif below:

Conclusion

If you have some HTML, CSS, and JavaScript knowledge, you can easily build Chrome extensions. I hope after reading this blog post, you will create some cool extensions.

CSV File :

A Comma-Separated Values (CSV) file is a plain text file containing a data list. These files are often used for the exchange of data between different applications. For example, databases and contact managers primarily support CSV files.

These files can sometimes be called character-separated values or comma-delimited files. They often use the comma character to separate data, but occasionally use other characters such as semicolons. The idea is that we can export the complex data from one application to a CSV file, and then import the data in that CSV file to another application.

Creation A text file in which a comma separates the value in a column is known as a CSV file. Let's start by creating a CSV file with the help of the data, which is mentioned below by saving with the .csv extension using the save As All files(.) option in the notepad.

Reading R has a rich set of functions. R provides the read.csv() function, which allows us to read a CSV file available in our current working directory. This function takes the file name as an input and returns all the records present on it.

Syntax

read.csv(filename, header = FALSE, sep = "")

Let's use our File.csv file to read records from it using the read.csv() function.

Example

data <- read.csv("File.csv")  
print(data)

Analysing

By default, the read.csv() function gives the output as a data frame. This can be easily checked as follows. Also, we can check the number of columns and rows.

Excel File :

The xlsx is a file extension of a spreadsheet file format that was created by Microsoft to work with Microsoft Excel. In the present era, Microsoft Excel is a widely used spreadsheet program that stores data in the .xls or .xlsx format. R allows us to read data directly from these files by providing some excel specific packages.

Reading

Like the CSV file, we can read data from an excel file. R provides the read.xlsx() function, which takes two arguments as input, i.e., file name and index of the sheet. This function returns the excel data in the form of a data frame in the R environment. There is the following syntax of the read.xlsx() function:

Syntax

read.xlsx(file_name,sheet_index)

R - Binary Files :

A binary file is a file that contains information stored only in form of bits and bytes. (0’s and 1’s). They are not humanly readable as the bytes in them translate to characters and symbols which contain many other non-printable characters. Attempting to read a binary file using any text editor will show characters like Ø and ð.

Sometimes, the data generated by other programs are required to be processed by R as a binary file. Also, R is required to create binary files which can be shared with other programs.

R has two functions WriteBin() and readBin() to create and read binary files.

Syntax

writeBin(object, con)

readBin(con, what, n )

Following is the description of the parameters used −

• con is the connection object to read or write the binary file.

• the object is the binary file in which it is written.

• what is the mode like character, integer, etc. representing the bytes to be read?

• n is the number of bytes to read from the binary file.

R - XML Files :

XML is a file format that shares both the file format and the data on the World Wide Web, intranets, and elsewhere using standard ASCII text. It stands for Extensible Markup Language (XML). Similar to HTML it contains markup tags. But unlike HTML where the markup tag describes the structure of the page, in XML the markup tags describe the meaning of the data contained in the file.

You can read an XML file in R using the "XML" package. This package can be installed using the following command.

install.packages("XML")

Reading XML File

The XML file is read by R using the function xmlParse(). It is stored as a list in R.

R - JSON Files :

JSON file stores data as text in a human-readable format. JSON stands for JavaScript Object Notation. R can read JSON files using the R-JSON package.

Install rjson Package In the R console, you can issue the following command to install the R-JSON package.

install.packages("rjson")

R - Web Data :

Using R programs, we can programmatically extract specific data from such websites. Some packages in R which are used to scrap data from the web are − "RCurl",XML+ 33 ", and "stringr". They are used to connect to the URLs+ required links for the files and download them to the local environment.

We will use the function getHTMLLinks---+() to gather the URLs of the files. Then we will use the function download.file() to save the files to the local system. As we will be applying the same code again and again for multiple files, we will create a function to be called multiple times. The filenames are passed as parameters in form of an R list object to this function.

This is about R programming. We will look into projects based on R in future blogs. Follow us for more interesting blogs. Subscribe to our newsletter.

1. Length Property:

Before moving to methods in the array, we have a property to find the length of the array(i.e., how many elements it consists of).

Let us understand by an example,

let arr = ["Apple", "Orange", "Banana"];
console.log(arr.length); //3

In the above example, we are creating an array with 3 elements, and with the help of the length property we can find out how many elements are there in the array. In this example, we have 3 elements inside the array and we got that as an output.

2. tostring():

tostring() function is used to convert the array into a string with separated by a comma.

Example:

let arr = ["Apple", "Orange", "Banana"];
console.log(arr.toString()); //Apple,Orange,Banana

3. join():

join() function is also used to convert the array into a string. By default in toString() it will use the comma separator to separate the string elements while converting it. Likewise, in join() the change is, that we will specify the separator as a parameter inside the join() function within double quotes(i.e., which separator we are going to use).

Let us understand by an example,

let arr = ["Apple", "Orange", "Banana"];
console.log(arr.join("|"));

In the above example, we are converting an array into a string and then we are joining the string elements using the separator(i.e., | ). We can use any separator as a parameter to join the string elements.

4. How to add and remove an element:

To add and remove elements from the array we will use the push() function and pop() function.

push() method:

To add an element to an array, we use the push() function.

Let us understand by an example,

let arr = ["Apple", "Orange", "Banana"];
arr.push("Grapes");
console.log(arr); //[ 'Apple', 'Orange', 'Banana', 'Grapes' ]

In the above example, the push() function will add the new element at the end of the array and make the changes in the existing array and display the output.

pop() method:

To remove an element from the array, we use the pop() function.

Let us understand by an example,

let arr = ["Apple", "Orange", "Banana"];
arr.pop();
console.log(arr); //[ 'Apple', 'Orange', 'Banana']

In the above example, pop() will remove the last element from the array and make the changes in the existing array and display the output.

5. shift() and unshift():

shift() and unshift() function is used to add and remove array elements from the beginning of the array.

unshift() method:

To add an element at the beginning of the array, we use the unshift() method.

Example:

let arr = ["Apple", "Orange", "Banana"];
arr.unshift("Grapes");
console.log(arr); //[ 'Grapes', 'Apple', 'Orange', 'Banana' ]

shift() method:

To remove an element at the beginning of the array, we use the shift() method.

Example:

let arr = ["Apple", "Orange", "Banana"];
arr.shift();
console.log(arr); //[ 'Orange', 'Banana' ]

6. slice():

The Slice() method is used to slice the array of elements. It has two parameters, are

• The first parameter is the starting index value.
• The second parameter is the ending index value.

Example:

let arr = ["Apple", "Orange", "Banana", "Grapes"];
output_arr = arr.slice(1,3);
console.log(output_arr); //[ 'Orange', 'Banana' ]

In the above example, the slice() method will slice the array of elements. Array elements are sliced by their index value. The first parameter index value is included and the second parameter index value is excluded(i.e., it represents the index value to slice the array).

7. splice():

The splice() method is also used to replace and remove the element from an array and display that array(i.e., it will not create a separate array for output).

The splice() method has three parameters are

• The First parameter is the starting index value.

• The second parameter is the count of the element from the start of the index value.

• The third parameter is the value we need to replace it with.

Example:

let arr = ["Apple", "Orange", "Banana", "Grapes"];
arr.splice(1,2,"Watermelon");
console.log(arr); //[ 'Apple', 'Watermelon', 'Grapes' ]

In the above example, we are starting with the index value 1 and we are replacing watermelon with the elements that count two elements from the index value 1.

8. concat():

concat() method will concat two arrays.

Example:

let X = ["Apple", "Orange", "Banana", "Grapes"];
let Y = ["Watermelon", "pineapple", "strawberry"];
console.log(X.concat(Y)); //[ 'Apple',  'Orange', 'Banana', 'Grapes',  'Watermelon' 'pineapple',  'strawberry' ]

In the above example, we are concatenating two arrays named X and Y.

In the next blog, we will explore sorting and searching methods in the array.

Previous blog, We learned about KNN(K-Nearest Neighbor). KNN is a classification algorithm based on classifying data points by determining similarity among existing data points. Now we are going to learn about self-learning algorithms.

Reinforcement Learning

As a machine learning technique, reinforcement learning concerns the appropriate actions that software agents should do in a given environment. The deep learning method includes reinforcement learning, which enables you to maximise a percentage of the cumulative reward.

Algorithm

Three approaches to implementing a Reinforcement Learning algorithm:

1. Value-Based
2. Policy-Based
3. Model Based

1. Value-Based:

Developers should aim to maximize the value function V in a value-based reinforcement learning strategy (s). With this approach, the agency anticipates that the current policy-covered states will eventually return.

2. Policy-Based:

In a policy-based RL technique, developers work to create a policy where every action you do now helps you reap the most rewards tomorrow.

Two policy-Based methods:-

Deterministic=> The policy π results in the same action for any state.

Stochastic=> The probability of any action is defined.

3. Model-Based:

For each environment in this Reinforcement Learning technique, a virtual model must be built. The agent gains the necessary skills to function in that environment.

Types of Reinforcement Learning

Positive Reinforcement:

It is described as an occurrence that results from a particular action. It strengthens and repeats the behaviour more frequently, which has a favourable effect on the agent's action.

This kind of reinforcement enables you to achieve your full potential and maintain change over a longer time frame. However, excessive reinforcement may result in state over-optimization, which might have an impact on the outcomes.

Negative Reinforcement:

A negative condition that should have been halted or avoided might reinforce a behaviour. This is known as "negative reinforcement." You can use it to specify the minimum performance standard. The disadvantage of this approach is that it just offers enough to satisfy the minimum behaviour.

Markov Decision Process

The reinforcement learning issues are formalised using the Markov Decision Process, or MDP. If the environment is entirely observable, a Markov Process can be used to model the environment's dynamic. In MDP, the agent continuously engages with the environment and takes action. The environment reacts to each action and creates a new state.

The RL environment is described using MDP, and practically all RL problems may be formalised using MDP.

MDP has 4 parameters:

• set of finite States S
• set of finite Actions A
• Rewards are obtained following the change from state S to state S' as a result of action a.
• Probability Pa

Q-Learning

A well-liked model-free reinforcement learning algorithm built on the Bellman equation is Q-learning. Learning the policy that can tell the agent what actions to perform to maximize the reward under what conditions is the basic goal of Q-learning. It is an off-policy RL that looks for the best course of action given the circumstances. In Q-learning, the agent's objective is to maximize Q's value. The Bellman equation can be used to determine the worth of Q-learning. Consider the following Bellman equation:

Process:

Temporal difference learning is accomplished using the Off policy RL algorithm Q-learning. Comparing temporally successive predictions are done using temporal difference learning methods. It learns the value function Q (S, a), which denotes the optimal course of action for a given state "s."

Applications of Reinforcement Learning

• Robotics
• Business strategy planning
• Aircraft control and robot motion control, etc..

Conclusion

I anticipate that this article will clearly explain the reinforcement learning algorithm. To ascertain more about machine learning and to do working machine learning models, check out my machine learning series. If you have any questions or doubts, mention them in this article's comments section, and connect with me to learn more about machine learning.

Learning is an interesting habit..!

An array is a special object in Javascript, that stores multiple values inside a single variable.

Why is Array needed?

Let’s take a scenario where we need to store multiple variables ranging up to 100(i.e., we need to keep 100 variables). Usually, what we will

Var a = 10;
Var b =30;
Var c = 40;
.
.
.
.
.
.
.
.
.
.
.
It goes on

In the above case, there is a lot of memory space is wasted and it’s not possible to store 100 variables in the above manner it creates issues in dynamic data storage also. So, the Array concept gives a solution to this problem. Because with the array we can store multiple values inside a single variable.

How to create an Array?

We use Square brackets(i.e., []) to enclose the array of elements inside a single variable.

Example:-

let arr = [ 1, 2, 3, 6,7 ];

How to access array elements?

We can access an array of elements by using its index value.

Let’s take an example,

let arr = [1,2,3,4,5];

In the above example, if we need to access the fourth element of the array. We will access it by its index value.

The Index value starts with 0 and it goes on.

The above picture shows the array of elements with its index value.

So, if we need to access fourth element of the array. We will do it by

console.log(arr[3]); // 4

Methods for defining array:

We can define an array in two methods, are

The first method is the regular method to define an array,

let arr = [ 1, 2, 3, 4 ];

The Second method is,

Step-1: First declare an empty array.

let arr = [];

Step-2: Next, using its index value - insert elements one after another.

arr[0] = “Banana”;
arr[1] = “Apple”;
arr[2] = “orange”;
console.log(arr); // [ “Banana”, “Apple”, “orange”];

In the next blog, we will explore array properties and methods.

What is a Package...?

A package is an appropriate way to organize the work and share it with others. Typically, a package will include code, documentation for the package and the functions inside, some tests to check everything works as it should, and data sets.

R packages are a collection of R functions, complied code and sample data. They are stored under a directory called "library" in the R environment. By default, R installs a set of packages during installation. More packages are added later when they are needed for some specific purpose. When we start the R console, only the default packages are available by default. Other packages which are already installed have to be loaded explicitly to be used by the R program that is going to use them.

All the packages available in the R language are listed at R Packages.

What are Repositaries...?

A repository is a place where packages are located and stored so you can install packages from it. Organizations and Developers have a local repository, typically they are online and accessible to everyone.

Installation of R Packages...?

There are two ways to add new R packages. One is installing directly from the CRAN directory and another is downloading the package to your local system and installing it manually.

Install Packages directly from CRAN

The commands used to install the packages directly from CRAN are given below

Syntax:

install.packages("package name")

Example

install.packages("XML")

Install Packages manually

Search for particular R packages on the web and download the required packages as .zip files in the required location in the local system.

install.packages(file_name_with_path, repos = NULL, type = "source")


install.packages("E:/XML_3.98-1.3.zip", repos = NULL, type = "source")

How to Load Packages in R Programming Language

Before a package can be used in the code, it must be loaded to the current R environment. You also need to load a package that is already installed previously but not available in the current environment.

A package is loaded using the following command −

library("package Name", lib.loc = "path to library")


install.packages("E:/XML_3.98-1.3.zip", repos = NULL, type = "source")

Difference Between a Package and a Library

library():

It is the command used to load a package, and it refers to the place where the package is contained, usually a folder on our computer.

Package:

It is a collection of functions bundled conveniently. The package is an appropriate way to organize our own work and share it with others.

How to Load More Than One Package at a Time

We can just input a vector of names to the install.packages() function to install a package, in the case of the library() function, this is not possible. We can load a set of packages one at a time, or if you prefer, use one of the many workarounds developed by R users.

To unload a given package, use the detach() function. The use will be:

detach("package:babynames", unload = TRUE)

A Short note on how to select the right packages

The most basic way to determine which package we need is by learning the R programming language. The first alternative can be to browse categories of CRAN packages. CRAN is the official repository, and also gives us the option to browse through packages.

Another alternative to finding packages can be R Documentation, a help documentation aggregator for R packages from CRAN, BioConductor, and GitHub, which offers you a search box ready for your requests directly on the main page.

Data Handling and Data Interfaces

In R, we can read data from files stored outside the R environment. We can also write data into files which will be stored and accessed by the operating system. R can read and write into various file formats like csv, excel, xml etc. R allows its users to work smoothly with the systems directories with the help of some pre-defined functions that take the path of the directory as the argument or return the path of the current directory that the user is working on. Below are some directory functions in R:

1. getwd(): This function is used to get the current working directory used by R.

2. setwd(): This function in R is used to change the path of the current working directory and the path of the directory is passed as an argument in the function.

3. list.files(): This function lists all files and folders present in the current working directory.

In the next blog, we will know about various data interfaces in R. Follow Quasar Community blogs for more interesting technical blogs.

The k-nearest neighbours (KNN) technique calculates the likelihood that a data point will belong to one group or another based on which group the data points closest to it do.

The k-nearest neighbour algorithm is an example of a supervised machine learning technique used to solve classification and regression problems. Its primary application, however, is in classification issues.

KNN is a slow, non-parametric learning method. It is known as a lazy learning algorithm or lazy learner because it does not perform any training when you provide the training data. During the training period, it does not perform any calculations and simply stores the data. A model is not created before a query is run on the dataset.

As a result, KNN is ideal for data mining. It is classified as a non-parametric method because it makes no assumptions about the underlying data distribution. KNN, in short, seeks to identify the group to which a data point belongs by examining the data points around it.

Think about the two groups, A and B. The algorithm examines the nearby data points' states to determine if a data point belongs to group A or group B. It is quite likely that the data point in question is in group A if the bulk of the data points is in group A, and vice versa. KNN also called the closest neighbour, is a technique for categorizing data points by comparing them to their nearest annotated data point.

K-NN classification should not be confused with K-means clustering. A supervised classification system called KNN sorts new data points according to the nearby ones. K-means clustering, on the other hand, divides data into a K number of clusters and is an unsupervised clustering algorithm.

Algorithm

The KNN algorithm is implemented using programming languages like Python and R. The KNN pseudocode is as follows:

1.Fill the data up

2.Select K value

3.For every informational point in the data:

3.1.All training data samples' Euclidean distance should be determined.

3.2.Place the distances on a list that is ordered, then sort it.

3.3.Pick the first K items in the sorted list.

3.4.Based on the vast majority of classes contained in the chosen points, label the test point.

4.End

Why KNN?

A crucial issue in data science and machine learning is classification. One of the earliest and most precise algorithms for pattern categorization and regression models is the KNN.

The following are some applications for the k-nearest neighbour algorithm:

• Credit rating

• Loan approval

• Data preprocessing

• Pattern recognition

KNN can be also used in recommendation systems because it can help find users with similar characteristics. It can, for example, be used in an online video streaming platform to recommend content that a user is more likely to watch based on what similar users watch.

For image classification, the KNN algorithm is used in computer vision. It's helpful in a variety of computer vision applications because it can group similar data points, such as grouping cats together and dogs in a different class.

Advantage

• It is simple to understand and implement.
• It applies to both classification and regression problems.
• It is ideal for non-linear data because it makes no assumptions about the underlying data.
• It is naturally capable of handling multi-class cases.
• With enough representative data, it can perform well.

Disadvantage

• Because it stores all of the training data, the associated computation cost is high.
• High memory storage is required.
• K's value must be determined.
• If N is large, prediction takes a long time.
• Sensitive to irrelevant characteristics

Example

Image classification

import numpy as np // linear algebra
import pandas as pd // data processing
import matplotlib.pyplot as plt  // to plot image, graph
import time
%matplotlib inline

// dataset for digit (0-9)
from sklearn.datasets import load_digits

Load dataset

digits = load_digits()

digits.keys()
#dict_keys(['data', 'target', 'target_names', 'images', 'DESCR'])

// dataset description
digits.DESCR

Already processed images

digits.images[0]

'''array([[ 0.,  0.,  5., 13.,  9.,  1.,  0.,  0.],
       [ 0.,  0., 13., 15., 10., 15.,  5.,  0.],
       [ 0.,  3., 15.,  2.,  0., 11.,  8.,  0.],
       [ 0.,  4., 12.,  0.,  0.,  8.,  8.,  0.],
       [ 0.,  5.,  8.,  0.,  0.,  9.,  8.,  0.],
       [ 0.,  4., 11.,  0.,  1., 12.,  7.,  0.],
       [ 0.,  2., 14.,  5., 10., 12.,  0.,  0.],
       [ 0.,  0.,  6., 13., 10.,  0.,  0.,  0.]])'''

Predictors, independent variables, features

digits.data

'''array([[ 0.,  0.,  5., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ..., 10.,  0.,  0.],
       [ 0.,  0.,  0., ..., 16.,  9.,  0.],
       ...,
       [ 0.,  0.,  1., ...,  6.,  0.,  0.],
       [ 0.,  0.,  2., ..., 12.,  0.,  0.],
       [ 0.,  0., 10., ..., 12.,  1.,  0.]])'''

Target variable, class, dependent variable

digits.target

#array([0, 1, 2, ..., 8, 9, 8])

These 1797 images (8 by 8 for a dimension of 64)

print('Image Data Shape', digits.images.shape)
# Image Data Shape (1797, 8, 8)


# 1797 labels
print('Label Data Shape', digits.target.shape)
#Label Data Shape (1797,)

X = digits.images
plt.figure(figsize=(20,10))
columns = 5
for i in range(5):
    plt.subplot(5 / columns + 1, columns, i + 1)
    plt.imshow(X[i],cmap=plt.cm.gray_r,interpolation='nearest')

from sklearn.metrics import accuracy_score,confusion_matrix # metrics error
from sklearn.model_selection import train_test_split # resampling method
X = digits.data
y = digits.target

Since its a multi-class prediction, to prevent error, we need some library

from sklearn.multiclass import OneVsRestClassifier
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.neighbors import KNeighborsClassifier
knn = OneVsRestClassifier(KNeighborsClassifier())
knn.fit(X_train,y_train)
//OneVsRestClassifier(estimator=KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
 //          metric_params=None, n_jobs=None, n_neighbors=5, p=2,
  //         weights='uniform'),
      //    n_jobs=None)

Predict for one observation

knn.predict(X_test[0].reshape(1,-1))
//array([2])

Predict for multiple observations (images) at once

knn.predict(X_test[0:10])
//array([2, 8, 2, 6, 6, 7, 1, 9, 8, 5])

Make predictions on the entire test data

predictions = knn.predict(X_test)
%time
# 98%
print('KNN Accuracy: %.3f' % accuracy_score(y_test,predictions))
//CPU times: user 0 ns, sys: 0 ns, total: 0 ns
//Wall time: 10 µs
//KNN Accuracy: 0.980

To create nice confusion metrics

import seaborn as sns
cm = confusion_matrix(y_test,predictions)
plt.figure(figsize=(9,9))
sns.heatmap(cm,annot=True, fmt='.3f', linewidths=.5, square=True,cmap='Blues_r')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
all_sample_title = 'Accuracy Score: {0}'.format(accuracy_score(y_test,predictions))
plt.title(all_sample_title,size=15)
#Text(0.5, 1.0, 'Accuracy Score: 0.98')

Conclusion

This article clearly explains the process of image classification using KNN in the machine learning model. To learn more about machine learning and how to make machine learning models, check out my machine learning series. If you have any questions or doubts, mention them in this article's comments section, and connect with me to learn more about machine learning.

Learning is an interesting habit..!

Chrome is the most used browsing application. Many people love it many people hate it but it can get the job done. But because of its easy-to-use functionalities, it is very easy to get distracted. In this blog, we are going to see 10 chrome extensions that can boost your productivity and keep you focused on work and less distracted

Extensions

1,Unhook

link:- unhook

Many people prefer to learn on youtube because of how convenient it is to learn for free also you wide variety of choices to choose from But it has its disadvantages. Youtube knows what you like and what kind of videos you click on the most. So the recommendation system is based on what you like not what you want. It is very easy to get distracted, whether a video from your favorite content creator or a cat video. So if you want to learn it is very difficult to ignore distractions. Unhook helps you to stop getting distracted by the recommended videos by simply hiding the recommended videos. You can even choose to hide comments from the videos. By using unhook you can save 5 to 50 minutes of short bust minutes that are wasted due to recommended videos and other distractions

2,Idenati

link:- Idenati

While writing this blog I had a lot of issues in the opening hash node itself cause I don't even remember the password for my account cause I have a ton of different accounts in different domains so it is very frustrating to remember every password for every account. That's where Idenati comes in handy. In Identity, you can have every account in one dashboard where you have to enter your password and email or username and you can save it right. Next time when you want to open a website then it will be right there just a click no hassle at all.

3,Forest

link:- Forest

Pomodoros are helpful to get into a flow state where you will do most of the deep work for starters forest is a great Pomodoro app where you can select how much you need each Pomodoro session to be. I use Pomodoro just to get started and then I will be in the flow state and start to work deeply you can use whichever way you like and for every session, you complete in the forest it will grow trees in the imaginary forest after completing a day you can see how many trees have grown in the forest it will give you a visual representation of how productive you are

4,loom

link:- loom

Loom is a screen recorder that is very convenient to use and You don't have to install a separate app to get small videos about a website or application. Loom allows you to take up to 5 mins of screen recording for free. You can also give a voice-over to recording with the camera. You can also share the direction with a link you don't need to download the recorded video which is very useful in sending videos about applications or websites to the team.

5,Calendly

link:- Calendly

Schedule meetings without the hassle. Never get double booked. Calendly works with your calendar to automate appointment scheduling. Once you have a Calendly account (free and paid plans), use the Calendly for Chrome extension to:

• Access all of your event types from anywhere on the web
• Leverage Calendly from the Gmail compose toolbar and add times to your email
• Leverage Calendly in LinkedIn Messaging, Recruiter, and Sales Navigator
• Select and share your availability in a few clicks
• Create one-off meetings to handpick times for VIPs
• Create meeting polls to find and book the best time for everyone to meet
• Mark and save event types as your favorites for even quicker access
• Add notes to your event types that only you and your teammates can see

Calendly connects with your personal and business calendars to save you time and simplify the tasks involved in meeting with people outside your company. Now you can connect and agree with your invitees on the best time to meet and schedule your meetings without the back-and-forth emails.

6, Angry study helper

link:- Angry study helper

An extension that gets grumpy when you open a new tab while studying. An extension that will get mad at you when you open a new tab while studying/working. Will give you the passion, vision, and aggression to grind through all your tasks.

How to use:

• Make sure to open all the tabs you need for your work before you turn on the extension.
• Turn on the extension and enjoy :)

7, Todolist

link:- Todolist

Focus To-Do combines Pomodoro Timer with Task Management, it is a science-based app that will motivate you to stay focused and get things done.

It brings Pomodoro Technique and Does List into one place, you can capture and organize tasks into your to-do lists, start a focus timer and focus on work & study, set reminders for important tasks and errands, and check the time spent at work.

It's the ultimate app for managing Tasks, Reminders, Lists, Calendar events, Grocery lists, and checklists, helping you focus on work & study and tracking your working hours.

Focus To-Do syncs between your phone and computer, so you can access your lists from anywhere.

8, Readwise

link:- Readwise

Create highlights from any webpage and sync directly to your Readwise account. Readwise Highlighter (Community) is a simple highlight creator extension for your Readwise account. Now you are not restricted by highlight sources Readwise supports syncing, and can create highlights from any online webpage text you want.

Simply do the following to create a new highlight and send it automatically to your Readwise account:

1. Install the extension.
2. Click on the extension in the browser bar, enter your Readwise access token (use the link on the extension to get your token), and click save.
3. Highlight any text on any webpage and right-click the highlighted area.
4. Select one of the menu options to automatically send the highlighted text to your Readwise account as a highlight.

9, Working

Link:- workona

Organize your work in the browser. Working helps you get your work under control and finally feel organized. Use it to manage tabs, organize projects, and bring together all of your work in the browser.

FEATURES

Workspaces Workspaces organize your work by project, so staying focused is easy. Within a workspace, you can keep your current work open as tabs, or save everything for a project as resources.

Tab manager Keep your tabs under control with tab management that’s built for working in the cloud. Suspend tabs, rearrange tabs with drag and drop, and rest easy knowing that every tab is automatically saved.

Secure backups You never have to worry about losing work, because Workona auto-saves all of your tabs. In case of a browser crash or other emergency, you can easily restore your tabs to get you back on track.

Access apps Use your favorite cloud apps within Working. Access your recent Google Docs, Asana Projects, Zoom Meetings, and hundreds of other cloud resources, right from Working. You can even create new cloud resources from the search bar.

Suspend tabs Dramatically reduce memory usage and keep your computer running faster. Our tab suspension helps you open workspaces and switch between them almost instantly.

Bookmark tabs Bookmark tabs in workspaces by adding them as resources. You can also choose to save an entire window as a workspace.

Search tabs Our powerful search helps you find your tabs and docs faster. Search all open tabs—or previously open tabs—at a moment’s notice.

Sync tabs Sync your tabs between computers to keep your work organized, no matter how many devices you use or where they are.

10, Audiblogs

Link:- Audiblogs

Instantly listen to any web page in the web browser or use a podcast player. Send As Podcast turns any web article, PDF, or email into a podcast using the latest AI narrator. You can listen in your podcast player, in the web browser, in the Chrome extension, or using our iOS/Android app.

• It sounds "scarily good." (demo at https://audioread.com)
• Your own, private podcast channel.

We will discuss the S3 and S4 classes, inheritance in these classes, and the methods provided by these classes.

S3

In oops, the S3 is used to overload any function. So that we can call the functions with different names and it depends on the type of input parameter or the number of parameters.

S4

S4 is the most important characteristic of oops. However, this is a limitation, as it is quite difficult to debug. There is an optional reference class for S4.

Class and Object

In R, everything is an object. Therefore, programmers perform the OOPS concept when they write code in R. An object is a data structure that has some methods that can act upon its attributes. An object is a data structure that contains some methods that act upon its attributes.

In R, classes are the outline or design for the object. Classes encapsulate the data members, along with the functions. In R, there are two most important classes, i.e., S3 and S4, which play an important role in performing OOPs concepts.

1. S3 Class : With S3 class, we can take advantage of the ability to implement the generic function. Furthermore, using only the first argument, S3 is capable of dispatching. S3 differs from traditional programming languages such as Java, C ++, and C #, which implement OO passing messages. This makes S3 easy to implement. In the S3 class, the generic function calls the method. S3 is very casual and has no formal definition of classes.

To create an object for this class we will create a list that will contain all the class members. Then this list is passed to the class() method as an argument.

Syntax:

var_name <- list(member1, member2, member3.........memberN)

Example:

var < - list(name="Hari", Reg_No=20)

// Defining Class

class(var) < - "Student" 

// Object creation

var 

Output
//$name
//[1] "Hari"

//$Reg_No
//[1] 20

//attr(, "class")
//[1] "Student"

Inheritance in S3 Class Inheritance is an important concept in OOP(object-oriented programming) which allows one class to derive the features and functionalities of another class. This feature facilitates code-reusability. Inheritance means extracting the features of one class into another class. In the S3 class of R, inheritance is achieved by applying the class attribute in a vector. S3 class objects inherit only methods from their base class.

Example:

student < - function(k, l) {
    value < - list(name=k, Roll=l)
    attr(value, "class") < - "student"
    value
}

print.student < - function(obj) {


    cat("Name:", obj$name, "\n")
    cat("Roll", obj$roll, "\n")}

s < - list(name="Hari", Roll=21, country="India")


class(s) < - c("Student", "student")
s

Output
//Name: Hari
//Roll: 21

2. S4 Class The S4 class is similar to the S3 but more formal than the latter. It differs from S3 in two different ways. S4 class has a predefined definition. It contains functions for defining methods and generics. It makes multiple dispatches easy. This class contains auxiliary functions for defining methods and generics.

In R, we use setClass() command for creating S4 class. In S4 class, we will specify a function for verifying the data consistency and also specify the default value. In R, member variables are called slots.

Syntax setClass(“myclass”, slots=list(name=”character”, Roll_No=”numeric”))

Example

setClass("Student", slots=list(name="character",
                               Roll_No="numeric"))

a <- new("Student", name="Hari", Roll_No=20)

a

Output
//Slot "name":
// "Hari"

//Slot "Roll_No":
//20

Inheritance in S4 Class

Like S3 class, we can perform inheritance in the S4 class also. The derived class will inherit both attributes and methods of the parent class. Let's start understanding how we can perform inheritance in the S4 class. There are the following ways to perform inheritance in the S4 class:

Example:

setClass("student",
         slots=list(name="character",
                    age="numeric", rno="numeric")
         )


setMethod("show", "student",
          function(obj){
              cat(obj@name, "\n")
              cat(obj@age, "\n")
              cat(obj@rno, "\n")
          }
          )


setClass("InternationalStudent",
         slots=list(country="character"),
         contains="student"
         )


s < - new("InternationalStudent", name="Hari",
          age=22, rno=15, country="India")
show(s)

Output
//Hari
//22 
//15

That is it for today, we can look into libraries in R in the next blog. Subscribe to our newsletter for more interesting blogs.

The machine learning model is a sector which can be trained and tested with specific data to recognise the pattern. we can prepare a model with selected datasets to perform a particular task. Our developer datasets well practise the model in the development stage. Then the model is evaluated by the client's dataset. The quality of the model will be based on the result's accuracy. In other words, the result explains the functionality and how deeply the model works on the datasets.

Why do we need Machine Learning Models?

Machine Learning Models are like a brain part of Machine learning. Without brains we can't do anything like that so without models machine learning cannot be implemented.

Example

A baby doesn't know how a cat and dog look. While growing up, the child will recognise the animal and say it is a cow and it is a dog by its characteristics. Here, Cat and Dog are the object and its characteristics are the input data. Whether, the object much similarly behave like cat's characteristics then the model confirms the object is cat. This kind of analysing and evaluating process is done on Machine Learning model.

Classification of Models

There are three learning models for algorithms that are based on various business objectives and data sources. Every machine learning algorithm gravitates toward one of the following three models:

• Supervised Learning

• Unsupervised Learning

• Reinforcement Learning

Workflow of Model

1-Explore & Process Data:

You are aware that machines initially learn from the information you provide them with. It is crucial to gather trustworthy data so that your machine learning model can identify the proper patterns. How accurate your model is will depend on the quality of the data you provide the computer. Inaccurate or out-of-date data will result in inaccurate results or predictions that are irrelevant.

As it will directly impact the result of your model, be sure to obtain data from a reputable source. Good data is pertinent, has few duplicated and missing information, and accurately represents all of the classifications and subcategories that are there. You must arrange your data after you receive it. This is possible by

Combining and randomising all of the data you have. This makes sure that the distribution of the data is fair and that the learning process is unaffected by the ordering. Cleaning the data to get rid of unnecessary information, missing values, duplicate values, rows, and columns, data type conversion, etc. The dataset's rows, columns, and index of rows and columns may even need to be changed. Visualize the data to comprehend its organisation and the connections between the many variables and classes it contains. Dividing the cleaned data into two sets, one for training and the other for testing. The set that your model learns from is called the training set. After training, your model's accuracy is tested using a testing set.

2-Modelling:

The most crucial phase of machine learning is training. To detect patterns and generate predictions, you feed your machine learning model the prepared data during training. As a result, the model gains knowledge from the data to complete the given task. The model improves in prediction over time with training.

You must evaluate your model's performance after training it. To do this, the model's performance is evaluated using data that has never been seen before. The testing set into which you previously divided our data is the unseen data used. The model is already accustomed to the data and detects the same patterns in it as it did during training, therefore using the same data for testing will result in an inaccurate measurement. You will receive excessively high accuracy as a result. You can accurately gauge the performance and speed of your model by using testing data.

3-Deployment:

Trained and Evaluated models outcome is deployed to production. Deployed production is monitored and altered and re-do the steps again.

Conclusion

I hope this article clearly explained the process of creating a machine learning model. To learn more about machine learning and how to make machine learning models, check out my machine learning series. If you have any questions or doubts, mention them in this article's comments section, and connect with me to learn more about machine learning.

Learning is an interesting habit..!

In today’s blog, we will solve another three fundamental problems using Javascript.

Problem - 1: Filter the negative numbers in an array

Program Algorithm:

Step-1: Start the Program.

Step-2: Declare the array inside the variable name i.e., arr.

Step-3: we will use a Javascript in-built filter() function used to filter the array elements, which will filter the array of elements that satisfy the particular condition(i.e., it should display only the array elements which are greater than -1.).

Step-4: Inside the filter() function, we will give the condition as the call-back function and it will filter out the array elements.

Step-5: The resultant will also be in an array and it is stored in the res_arr variable.

Step-6: End the Program.

Program Code:

let arr = [1,-8,5,6,-4,-2,-1,0,7,9,3];
let res_arr = arr.filter(
    function result(a){
        return a >-1
    }
    )
console.log(res_arr); //[ 1, 5, 6, 0, 7, 9, 3 ]

Problem - 2: Remove the spaces found in a string

Program Algorithm:

Step-1: Start the program.

Step-2: Declare the str variable with blank spaces in front and back of the string characters.

Step-3: we will use the inbuilt trim() function in Javascript to trim the empty blank spaces in a string.

Step-4: at last, we will store the output in the res_str variable.

Step-5: End the Program.

Program Code:

let str = "     I love Programming      ";
let res_str = str.trim(" ");
console.log(res_str);

Note:-

We will use the trim() while sending input data from the server to the database. For example, when a user needs to give his name in his/her input field named Name, if he/she fills it with spaces in front or back of the name. It will create an issue while storing the data in the database. So, if we use trim() - it automatically trims the empty space and sends the data to the database.

Problem - 3: Celsius to Fahrenheit and Fahrenheit to Celsius

Program Algorithm:

Step-1: Start the Program.

Step-2: To convert celsius to Fahrenheit we will use the formula(i.e., divide the celsius value by 1.8 and add 32 to it.

Step-3: Here, we will use the basic function concept to get the result and at last we will console the result.

Step-4: Procedure to convert Fahrenheit To Celsius and only the formula to convert varies. The formula is we will subtract the Fahrenheit value by 32 and divide it by 1.8.

Step-5: End the Program.

Program Code:

// celsiusToFahrenheit
function celsiusToFahrenheit(d)
{
    return d * 1.8 + 32;
}
let ctf_result = celsiusToFahrenheit(20);
console.log(ctf_result); //68

// fahrenheitToCelsius
function fahrenheitToCelsius(n)
{
    return (n - 32) / 1.8;
}
let ftc_result = fahrenheitToCelsius(68);
console.log(ftc_result); //20

In the next blog, we will explore arrays and their methods in Javascript.

Clustering is the process of dividing the dataset into groups, or clusters. The objective is to divide the data into clusters where points are very similar to one another and distinct from one another. Clustering algorithms assign (or predict) a number to each data point, indicating which cluster that specific point belongs to, similarly to classification algorithms.

K-Means Clustering

One of the simplest and most popular clustering techniques is k-means clustering. It looks for cluster centres that are typical of particular data regions. Two steps are alternated in the algorithm: Each cluster centre is established as the mean of the given data points after each data point is assigned to the closest cluster centre. When instances are assigned to clusters consistently, the method is complete.

Initially, define a target number K, which refers to the number of centroids you need in the dataset. A centroid is an imaginary or actual location representing the centre of the cluster. The K-means algorithm identifies the K number of centroids, then allocates every data point to the nearest cluster while keeping the centroids as small as possible. To process the learning data, the K-means algorithm in data mining starts with the first group of randomly selected centroids, which are used as the beginning points for every cluster, and then performs iterative (repetitive) calculations to optimize the positions of the centroids. It halts after creating and optimizing clusters when either the centroids have stabilized i.e. there is no change in their values because the clustering has been successful or the defined number of iterations has been achieved.

Algorithmic Steps for K-Means Clustering

Step 1 : Let X = {X1,X2,…,Xn} be the set of data points.

Step 2: Arbitrarily select ‘K’ cluster centres denoted as C1, C2,..., Ck.

Step 3: Calculate the distance between each data point with the cluster centres by using any distance measurers.

Step 4: Assign the data points to the cluster centre whose distance from the cluster the centre is minimum with respect to other cluster centres.

Step 5: Recalculate the distance between each data point and its centres to get new cluster centres using mean.

Step 6: Repeat from step 3 till there is no change in the cluster centre

Methods

Let's discuss the methods used to find the optimal number of k clusters

Elbow Method

The Elbow method looks at the total within-cluster sum of squares (WSS) as a function of the number of clusters. One should choose a number of clusters so that adding another cluster doesn’t improve much better the total WSS. The total WSS measures the compactness of the clustering and we want it to be as small as possible. WSS is the sum of squares used to determine the optimal value of K. The formula to find WSS is given as

Where pi represents the data point and q(i) represents the cluster centre.

Average Silhouette Method

The Silhouette method is used for the interpretation and validation of consistency within clusters of data. It provides a graphical representation of how well each object has been classified. It measures the quality of a clustering. That is, it determines how well each object lies within its cluster. A high average silhouette width indicates a good clustering. The average silhouette method computes the average silhouette of observations for different values of K. The optimal number of clusters K is the one that maximizes the average silhouette over a range of possible values for K.

Gap Statistic Method

The gap statistic approach can be applied to any clustering method. The gap statistic compares the total within intra-cluster variation for different values of K with their expected values under the null reference distribution of the data. The estimate of the optimal clusters will be the value that maximizes the gap statistic. This means that the clustering structure is far away from the random uniform distribution of points.

We will learn new things in the upcoming blog, until connect with me and share this blog with your friends and colleagues!....💖.

As we know Python is a suitable language for scriptwriters and developers. Let’s write a script for Voice Assistant using Python. The query for the assistant can be manipulated as per the user’s need. Speech recognition is the process of converting audio into text. This is commonly used in voice assistants like Alexa, Siri, etc. Python provides an API called SpeechRecognition to allow us to convert audio into text for further processing. In this article, we will look at converting large or long audio files into text using the SpeechRecognition API in python.

Modules Required

Subprocess:- This module is used to get system subprocess details used in various commands i.e Shutdown, Sleep, etc. This module comes built-in with Python.

WolframAlpha:- It is used to compute expert-level answers using Wolfram’s algorithms, knowledgebase, and AI technology. To install this module type the below command in the terminal.

Pyttsx3:- This module is used for the conversion of text to speech in a program it works offline. To install this module type the below command in the terminal.

Tkinter:- This module is used for building GUI and comes inbuilt with Python. This module comes built-in with Python.

Wikipedia:- As we all know Wikipedia is a great source of knowledge just like GeeksforGeeks we have used the Wikipedia module to get information from Wikipedia or to perform a Wikipedia search. To install this module type the below command in the terminal.

Speech Recognition:- Since we’re building an Application of voice assistant, one of the most important things in this is that your assistant recognizes your voice (means what you want to say/ ask). To install this module type the below command in the terminal.

Web browser:- To perform Web Search. This module comes built-in with Python.

Ecapture:- To capture images from your Camera. To install this module type the below command in the terminal.

Pyjokes:- Pyjokes is used for the collection of Python Jokes over the Internet. To install this module type the below command in the terminal.

Datetime:- Date and Time are used to showing Date and Time. This module comes built-in with Python.

Twilio:- Twilio is used for making calls and messages. To install this module type the below command in the terminal.

Requests: Requests are used for making GET and POST requests. To install this module type the below command in the terminal.

BeautifulSoup: Beautiful Soup is a library that makes it easy to scrape information from web pages. To install this module type the below command in the terminal.

Implementation

Import the below libraries

import subprocess
import wolframalpha
import pyttsx3
import tkinter
import json
import random
import operator
import speech_recognition as sr
import datetime
import wikipedia
import webbrowser
import os
import winshell
import pyjokes
import feedparser
import smtplib
import ctypes
import time
import requests
import shutil
from twilio.rest import Client
from clint.textui import progress
from ecapture import ecapture as ec
from bs4 import BeautifulSoup
import win32com.client as wincl
from urllib.request import urlopen

Now we will set our engine to Pyttsx3 which is used for text to speech in Python and sapi5 is a Microsoft speech application platform interface we will be using this for text to speech function.

engine = pyttsx3.init('sapi5')
voices = engine.getProperty('voices')
engine.setProperty('voice', voices[1].id)

You can change the voice Id to “0” for the Male voice while using assistant here we are using a Female voice for all text to speech

def speak(audio):
    engine.say(audio)
    engine.runAndWait()

def wishMe():
    hour = int(datetime.datetime.now().hour)
    if hour>= 0 and hour<12:
        speak("Good Morning Sir !")

    elif hour>= 12 and hour<18:
        speak("Good Afternoon Sir !")

    else:
        speak("Good Evening Sir !")

    assname =("Jarvis 1 point o")
    speak("I am your Assistant")
    speak(assname)


def username():
    speak("What should i call you sir")
    uname = takeCommand()
    speak("Welcome Mister")
    speak(uname)
    columns = shutil.get_terminal_size().columns

    print("#####################".center(columns))
    print("Welcome Mr.", uname.center(columns))
    print("#####################".center(columns))

    speak("How can i Help you, Sir")

def takeCommand():

    r = sr.Recognizer()

    with sr.Microphone() as source:

        print("Listening...")
        r.pause_threshold = 1
        audio = r.listen(source)

    try:
        print("Recognizing...")
        query = r.recognize_google(audio, language ='en-in')
        print(f"User said: {query}\n")

    except Exception as e:
        print(e)
        print("Unable to Recognize your voice.")
        return "None"

    return query

def sendEmail(to, content):
    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.ehlo()
    server.starttls()

    # Enable low security in gmail
    server.login('your email id', 'your email password')
    server.sendmail('your email id', to, content)
    server.close()

Main Function starts here, we will now call all these functions in the main function.

if __name__ == '__main__':
    clear = lambda: os.system('cls')

    # This Function will clean any
    # command before execution of this python file
    clear()
    wishMe()
    username()

    while True:

        query = takeCommand().lower()

        # All the commands said by user will be
        # stored here in 'query' and will be
        # converted to lower case for easily
        # recognition of command
        if 'wikipedia' in query:
            speak('Searching Wikipedia...')
            query = query.replace("wikipedia", "")
            results = wikipedia.summary(query, sentences = 3)
            speak("According to Wikipedia")
            print(results)
            speak(results)

        elif 'open youtube' in query:
            speak("Here you go to Youtube\n")
            webbrowser.open("youtube.com")

        elif 'open google' in query:
            speak("Here you go to Google\n")
            webbrowser.open("google.com")

        elif 'open stackoverflow' in query:
            speak("Here you go to Stack Over flow.Happy coding")
            webbrowser.open("stackoverflow.com")

        elif 'play music' in query or "play song" in query:
            speak("Here you go with music")
            # music_dir = "G:\\Song"
            music_dir = "C:\\Users\\GAURAV\\Music"
            songs = os.listdir(music_dir)
            print(songs)
            random = os.startfile(os.path.join(music_dir, songs[1]))

        elif 'the time' in query:
            strTime = datetime.datetime.now().strftime("% H:% M:% S")
            speak(f"Sir, the time is {strTime}")

        elif 'open opera' in query:
            codePath = r"C:\\Users\\GAURAV\\AppData\\Local\\Programs\\Opera\\launcher.exe"
            os.startfile(codePath)

        elif 'email to gaurav' in query:
            try:
                speak("What should I say?")
                content = takeCommand()
                to = "Receiver email address"
                sendEmail(to, content)
                speak("Email has been sent !")
            except Exception as e:
                print(e)
                speak("I am not able to send this email")

        elif 'send a mail' in query:
            try:
                speak("What should I say?")
                content = takeCommand()
                speak("whome should i send")
                to = input()
                sendEmail(to, content)
                speak("Email has been sent !")
            except Exception as e:
                print(e)
                speak("I am not able to send this email")

        elif 'how are you' in query:
            speak("I am fine, Thank you")
            speak("How are you, Sir")

        elif 'fine' in query or "good" in query:
            speak("It's good to know that your fine")

        elif "change my name to" in query:
            query = query.replace("change my name to", "")
            assname = query

        elif "change name" in query:
            speak("What would you like to call me, Sir ")
            assname = takeCommand()
            speak("Thanks for naming me")

        elif "what's your name" in query or "What is your name" in query:
            speak("My friends call me")
            speak(assname)
            print("My friends call me", assname)

        elif 'exit' in query:
            speak("Thanks for giving me your time")
            exit()

        elif "who made you" in query or "who created you" in query:
            speak("I have been created by Gaurav.")

        elif 'joke' in query:
            speak(pyjokes.get_joke())

        elif "calculate" in query:

            app_id = "Wolframalpha api id"
            client = wolframalpha.Client(app_id)
            indx = query.lower().split().index('calculate')
            query = query.split()[indx + 1:]
            res = client.query(' '.join(query))
            answer = next(res.results).text
            print("The answer is " + answer)
            speak("The answer is " + answer)

        elif 'search' in query or 'play' in query:

            query = query.replace("search", "")
            query = query.replace("play", "")        
            webbrowser.open(query)

        elif "who i am" in query:
            speak("If you talk then definitely your human.")

        elif "why you came to world" in query:
            speak("Thanks to Gaurav. further It's a secret")

        elif 'power point presentation' in query:
            speak("opening Power Point presentation")
            power = r"C:\\Users\\GAURAV\\Desktop\\Minor Project\\Presentation\\Voice Assistant.pptx"
            os.startfile(power)

        elif 'is love' in query:
            speak("It is 7th sense that destroy all other senses")

        elif "who are you" in query:
            speak("I am your virtual assistant created by Gaurav")

        elif 'reason for you' in query:
            speak("I was created as a Minor project by Mister Gaurav ")

        elif 'change background' in query:
            ctypes.windll.user32.SystemParametersInfoW(20,
                                                    0,
                                                    "Location of wallpaper",
                                                    0)
            speak("Background changed successfully")

        elif 'open bluestack' in query:
            appli = r"C:\\ProgramData\\BlueStacks\\Client\\Bluestacks.exe"
            os.startfile(appli)

        elif 'news' in query:

            try:
                jsonObj = urlopen('''https://newsapi.org / v1 / articles?source = the-times-of-india&sortBy = top&apiKey =\\times of India Api key\\''')
                data = json.load(jsonObj)
                i = 1

                speak('here are some top news from the times of india')
                print('''=============== TIMES OF INDIA ============'''+ '\n')

                for item in data['articles']:

                    print(str(i) + '. ' + item['title'] + '\n')
                    print(item['description'] + '\n')
                    speak(str(i) + '. ' + item['title'] + '\n')
                    i += 1
            except Exception as e:

                print(str(e))


        elif 'lock window' in query:
                speak("locking the device")
                ctypes.windll.user32.LockWorkStation()

        elif 'shutdown system' in query:
                speak("Hold On a Sec ! Your system is on its way to shut down")
                subprocess.call('shutdown / p /f')

        elif 'empty recycle bin' in query:
            winshell.recycle_bin().empty(confirm = False, show_progress = False, sound = True)
            speak("Recycle Bin Recycled")

        elif "don't listen" in query or "stop listening" in query:
            speak("for how much time you want to stop jarvis from listening commands")
            a = int(takeCommand())
            time.sleep(a)
            print(a)

        elif "where is" in query:
            query = query.replace("where is", "")
            location = query
            speak("User asked to Locate")
            speak(location)
            webbrowser.open("https://www.google.nl / maps / place/" + location + "")

        elif "camera" in query or "take a photo" in query:
            ec.capture(0, "Jarvis Camera ", "img.jpg")

        elif "restart" in query:
            subprocess.call(["shutdown", "/r"])

        elif "hibernate" in query or "sleep" in query:
            speak("Hibernating")
            subprocess.call("shutdown / h")

        elif "log off" in query or "sign out" in query:
            speak("Make sure all the application are closed before sign-out")
            time.sleep(5)
            subprocess.call(["shutdown", "/l"])

        elif "write a note" in query:
            speak("What should i write, sir")
            note = takeCommand()
            file = open('jarvis.txt', 'w')
            speak("Sir, Should i include date and time")
            snfm = takeCommand()
            if 'yes' in snfm or 'sure' in snfm:
                strTime = datetime.datetime.now().strftime("% H:% M:% S")
                file.write(strTime)
                file.write(" :- ")
                file.write(note)
            else:
                file.write(note)

        elif "show note" in query:
            speak("Showing Notes")
            file = open("jarvis.txt", "r")
            print(file.read())
            speak(file.read(6))

        elif "update assistant" in query:
            speak("After downloading file please replace this file with the downloaded one")
            url = '# url after uploading file'
            r = requests.get(url, stream = True)

            with open("Voice.py", "wb") as Pypdf:

                total_length = int(r.headers.get('content-length'))

                for ch in progress.bar(r.iter_content(chunk_size = 2391975),
                                    expected_size =(total_length / 1024) + 1):
                    if ch:
                    Pypdf.write(ch)

        # NPPR9-FWDCX-D2C8J-H872K-2YT43
        elif "jarvis" in query:

            wishMe()
            speak("Jarvis 1 point o in your service Mister")
            speak(assname)

        elif "weather" in query:

            # Google Open weather website
            # to get API of Open weather
            api_key = "Api key"
            base_url = "http://api.openweathermap.org / data / 2.5 / weather?"
            speak(" City name ")
            print("City name : ")
            city_name = takeCommand()
            complete_url = base_url + "appid =" + api_key + "&q =" + city_name
            response = requests.get(complete_url)
            x = response.json()

            if x["code"] != "404":
                y = x["main"]
                current_temperature = y["temp"]
                current_pressure = y["pressure"]
                current_humidiy = y["humidity"]
                z = x["weather"]
                weather_description = z[0]["description"]
                print(" Temperature (in kelvin unit) = " +str(current_temperature)+"\n atmospheric pressure (in hPa unit) ="+str(current_pressure) +"\n humidity (in percentage) = " +str(current_humidiy) +"\n description = " +str(weather_description))

            else:
                speak(" City Not Found ")

        elif "send message " in query:
                # You need to create an account on Twilio to use this service
                account_sid = 'Account Sid key'
                auth_token = 'Auth token'
                client = Client(account_sid, auth_token)

                message = client.messages \
                                .create(
                                    body = takeCommand(),
                                    from_='Sender No',
                                    to ='Receiver No'
                                )

                print(message.sid)

        elif "wikipedia" in query:
            webbrowser.open("wikipedia.com")

        elif "Good Morning" in query:
            speak("A warm" +query)
            speak("How are you Mister")
            speak(assname)

        # most asked question from google Assistant
        elif "will you be my gf" in query or "will you be my bf" in query:
            speak("I'm not sure about, may be you should give me some time")

        elif "how are you" in query:
            speak("I'm fine, glad you me that")

        elif "i love you" in query:
            speak("It's hard to understand")

        elif "what is" in query or "who is" in query:

            # Use the same API key
            # that we have generated earlier
            client = wolframalpha.Client("API_ID")
            res = client.query(query)

            try:
                print (next(res.results).text)
                speak (next(res.results).text)
            except StopIteration:
                print ("No results")

        # elif "" in query:
            # Command go here
            # For adding more commands

Outro

And that's how you create your own personal assistant using python. You can alter some commands as your wish and make them more personalized as your wish. Subscribe and follow to quasar community blogs for more tech content

So there are various types of data structures :

1. Vectors

Vectors are the most basic data structures in R. It consists of Basic data types of a given length. These are one-dimensional data structures. There are various operations we can perform in vectors.

Vector arithmetic:

Arithmetic operations like addition, subtraction, and multiplication can be performed and the result is obtained as a vector datatype.

var1 <- c(3,8,4,5,0,11)
var2 <- c(4,11,0,8,1,2)

// Vector addition.
add.result <- var1+var2 
print(add.result)

// Vector subtraction.
sub.result <- var1-var2 
print(sub.result)

// Vector multiplication.
multi.result <- var1*var2 
print(multi.result)

// Vector division.
divi.result <- var1/var2 
print(divi.result)

//output
7 19 4 13 1 13

-1 -3 4 -3 -1 9
12 88 0 40 0 22
0.7500000 0.7272727  0.6250000 0.0000000 5.5000000

Vector Element Recycling

vectors with an unequal number of elements can be solved using arithmetic operations. The elements of the smaller vector are recycled to complete the operations.

var1 <- c(3,8,4,5,0,11)
var2 <- c(4,11)

add.result <- var1+var2
print(add.result)

sub.result <- var1-var2
print(sub.result)

//output
7 19 8 16 4 22
-1 -3 0 -6 -4 0

2. Lists

A list is an ordered collection of objects. It consists of a mixture of various data types. Therefore, A list can be defined as a special type of vector in which each element can be a different type.

Manipulating List elements

We can add, delete and update list elements. But we can update any element.

list2 <- list(matrix(c(3,9,5,1,-2,8), nrow = 2), c("Jan","Feb","Mar"), list(3,4,5))
list2[4] <- “HELLO”
print(list2[4])

//output
"Hello"

3. Dataframes

A dataframe is a table or a two-dimensional array-like structure in which each column contains the value of one variable and each row contains one set of values from each column.

Example of Dataframe:

Name = c("Amiya", "Raj", "Asish")
Language = c("R", "Python", "Java")
Age = c(22, 25, 45)
df = data.frame(Name, Language, Age)
print(df)

//Output
   Name  Language  Age
1  Amiya        R          22
2  Raj         Python    25
3  Asish     Java        45

4. Matrices

A matrix is an arrangement of numbers in rows and columns. Rows are the horizontally aligned values and vertically aligned values are called columns.

To create a matrix in R you need to use the function called matrix.

A = matrix(

    c(1, 2, 3, 4, 5, 6, 7, 8, 9),


    nrow = 3, ncol = 3, 

    byrow = TRUE                            
)

print(A)

//output
1 2 3
4 5 6 
7 8 9

5. Arrays

Arrays are data structures which store data of more than two dimensions. An array is a collection of a similar data type with contiguous memory allocation. In R, an array is created with the help of array() function. This function takes a vector as an input and uses the value in the dim parameter to create an array.

A = array(

    c(1, 2, 3, 4, 5, 6, 7, 8),


    dim = c(2, 2, 2)                       
)

print(A)

//output
, , 1

   1     3
   2    4

, , 2

    5    7
    6    8

6. Factors

Factors are also data objects that are used to categorize the data and store it as levels. Factors can store both strings and integers. Columns have a limited number of unique values so factors are very useful in columns. They can store both strings and integers. They are useful to categorize unique values in columns like “TRUE” or “FALSE”, or “MALE” or “FEMALE”, etc. It is very useful in data analysis for statistical modelling.

Factors are created with the help of factor() function by taking a vector as an input parameter.

fac = factor(c("Male", "Female", "Male",
               "Male", "Female", "Male", "Female"))

print(fac)

//output

Male   Female Male   Male   Female Male   Female
Levels: Female Male

That is up for this blog. we can discuss oop's concept libraries in R in the upcoming blogs. Follow Quasar community for more and also do subscribe for our newsletter.

Previous blog, we learned how to predict the weather and if the weather is fit to play or not. Likely we discuss another interesting machine learning algorithm here.

SVM

Here we are going to introduce another approach to classification using a family of algorithms called support vector machines. They can work with both linear and non-linear scenarios, allowing high performance in many different contexts. Together with neural networks, SVMs probably represent the best choice for many tasks where it's not easy to find out a good separating hyperplane. For example, It takes the past data as an input and outputs a line or a hyper-plane which separates the mixed data into classes. If Past data contains both strawberry and apple information to recognize what apples and strawberries look like. That’s SVM in play. It analyses the data and classifies it into one of the two categories based on the labeled data it already has. It will sort the apples under the apple category and the strawberries under the strawberry category.

Linear support vector machines

The fundamental idea behind SVMs is best explained with some pictures.

The two classes can clearly be separated easily with a straight line (they are linearly separable). The left plot shows the decision boundaries of three possible linear classifiers. The model whose decision boundary is represented by the dashed line is so bad that it does not even separate the classes properly. The other two models work perfectly on this training set, but their decision boundaries come so close to the instances that these models will probably not perform as well on new instances. In contrast, the solid line in the plot on the right represents the decision boundary of an SVM classifier; this line not only separates the two classes but also stays as far away from the closest training instances as possible. You can think of an SVM classifier as fitting the widest possible street (represented by the parallel dashed lines) between the classes. This is called large margin classification.

What is Gamma?

Gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’. The gamma parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.

What is Regularization?

The Regularization parameter (often termed as C parameter in python’s sklearn library) tells the SVM optimization how much you want to avoid misclassifying each training example. For large values of C, the optimization will choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. Conversely, a very small value of C will cause the optimizer to look for a larger-margin separating hyperplane, even if that hyperplane misclassifies more points.

Kernel

SVM algorithms use a set of mathematical functions that are defined as the kernel. The function of kernel is to take data as input and transform it into the required form.

Example

Text Classification

Tool Used: Python IDE

Modules used: pandas,sklearn

import pandas as pd
import numpy as np
from nltk.tokenize import word_tokenize
from nltk import pos_tag
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from sklearn.preprocessing import LabelEncoder
from collections import defaultdict
from nltk.corpus import wordnet as wn
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import model_selection, naive_bayes, svm
from sklearn.metrics import accuracy_score

#Set Random seed
np.random.seed(500)

# Add the Data using pandas
Corpus = pd.read_csv(r"C:\Users\gunjit.bedi\Desktop\NLP Project\corpus_small.csv",encoding='latin-1')

# Step - 1: Data Pre-processing - This will help in getting better results through the classification algorithms

# Step - 1a : Remove blank rows if any.
Corpus['text'].dropna(inplace=True)

# Step - 1b : Change all the text to lower case. This is required as python interprets 'dog' and 'DOG' differently
Corpus['text'] = [entry.lower() for entry in Corpus['text']]

# Step - 1c : Tokenization : In this, each entry in the corpus will be broken into a set of words
Corpus['text']= [word_tokenize(entry) for entry in Corpus['text']]

# Step - 1d : Remove Stop words, Non-Numeric and perform Word Stemming/Lemmenting.

# WordNetLemmatizer requires Pos tags to understand if the word is noun or verb or adjective etc. By default it is set to Noun
tag_map = defaultdict(lambda : wn.NOUN)
tag_map['J'] = wn.ADJ
tag_map['V'] = wn.VERB
tag_map['R'] = wn.ADV


for index,entry in enumerate(Corpus['text']):
    # Declaring Empty List to store the words that follow the rules for this step
    Final_words = []
    # Initializing WordNetLemmatizer()
    word_Lemmatized = WordNetLemmatizer()
    # pos_tag function below will provide the 'tag' i.e if the word is Noun(N) or Verb(V) or something else.
    for word, tag in pos_tag(entry):
        # Below condition is to check for Stop words and consider only alphabets
        if word not in stopwords.words('english') and word.isalpha():
            word_Final = word_Lemmatized.lemmatize(word,tag_map[tag[0]])
            Final_words.append(word_Final)
    # The final processed set of words for each iteration will be stored in 'text_final'
    Corpus.loc[index,'text_final'] = str(Final_words)

#print(Corpus['text_final'].head())

# Step - 2: Split the model into Train and Test Data set
Train_X, Test_X, Train_Y, Test_Y = model_selection.train_test_split(Corpus['text_final'],Corpus['label'],test_size=0.3)

# Step - 3: Label encode the target variable  - This is done to transform Categorical data of string type in the data set into numerical values
Encoder = LabelEncoder()
Train_Y = Encoder.fit_transform(Train_Y)
Test_Y = Encoder.fit_transform(Test_Y)

# Step - 4: Vectorize the words by using TF-IDF Vectorizer - This is done to find how important a word in the document is in comparison to the corpus
Tfidf_vect = TfidfVectorizer(max_features=5000)
Tfidf_vect.fit(Corpus['text_final'])

Train_X_Tfidf = Tfidf_vect.transform(Train_X)
Test_X_Tfidf = Tfidf_vect.transform(Test_X)


# Step - 5: Now we can run different algorithms to classify out data and check for accuracy

# Classifier - Algorithm - Naive Bayes
# fit the training dataset on the classifier
Naive = naive_bayes.MultinomialNB()
Naive.fit(Train_X_Tfidf,Train_Y)

# predict the labels on validation dataset
predictions_NB = Naive.predict(Test_X_Tfidf)

# Use accuracy_score function to get the accuracy
print("Naive Bayes Accuracy Score -> ",accuracy_score(predictions_NB, Test_Y)*100)


# Classifier - Algorithm - SVM
# fit the training dataset on the classifier
SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto')
SVM.fit(Train_X_Tfidf,Train_Y)

# predict the labels on validation dataset
predictions_SVM = SVM.predict(Test_X_Tfidf)

# Use accuracy_score function to get the accuracy
print("SVM Accuracy Score -> ",accuracy_score(predictions_SVM, Test_Y)*100)

From the code we learned how to classify the text and find the accurracy using the dataset.

We will learn new things in the upcoming blog, until connect with me and share this blog with your friends and colleagues!....💖.